[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:28 dist-test-slave-9fh2 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:28 dist-test-slave-9fh2 krb5kdc[399](info): set up 2 sockets
Sep 27 00:52:28 dist-test-slave-9fh2 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 00:52:29.000434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:29.136346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:29.268154 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:29.268518 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:29.268563 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:29.272938 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:29.273026 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:29.273046 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:29.273062 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:29.273077 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:29.279302 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:29.280702 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:29.281983 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:52:29.293362 412 server_base.cc:1034] running on GCE node
W20260927 00:52:29.293516 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:29.293418 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:29.293394 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:29.294411 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:29.295962 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:29.297139 412 hybrid_clock.cc:648] HybridClock initialized: now 1790470349297077 us; error 80 us; skew 500 ppm
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:29.300274 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:29.301409 412 webserver.cc:460] Webserver started at http://127.0.98.62:45021/ using document root <none> and password file <none>
I20260927 00:52:29.302083 412 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:29.302147 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:29.302349 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:29.304917 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "74bba1bfe74ab82f24e5389032462875"
server_key_iv: "b50ac0c6384a9a8bcd55f1daab1707e2"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.305570 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "74bba1bfe74ab82f24e5389032462875"
server_key_iv: "b50ac0c6384a9a8bcd55f1daab1707e2"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.310498 412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:52:29.313844 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.315979 412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:29.316195 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "74bba1bfe74ab82f24e5389032462875"
server_key_iv: "b50ac0c6384a9a8bcd55f1daab1707e2"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.316491 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:29.342264 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:29.346464 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:29.346711 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:29.355757 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38599
I20260927 00:52:29.355792 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38599 every 8 connection(s)
I20260927 00:52:29.357060 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:29.360826 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:29.365187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 412
I20260927 00:52:29.365379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:29.366953 392 external_mini_cluster.cc:1428] Setting key 5e918b95cd6092050ecf12ba186c025f
I20260927 00:52:29.367654 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d: Bootstrap starting.
I20260927 00:52:29.370705 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:29.371722 480 log.cc:824] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:29.373903 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d: No bootstrap required, opened a new log
I20260927 00:52:29.378239 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38599 } }
I20260927 00:52:29.378552 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:29.378625 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9f13b0a1674c1a851d9c1080b0fa6d, State: Initialized, Role: FOLLOWER
I20260927 00:52:29.379245 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38599 } }
I20260927 00:52:29.379391 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:29.379463 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:29.379582 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:29.380548 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38599 } }
I20260927 00:52:29.380923 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9f13b0a1674c1a851d9c1080b0fa6d; no voters:
I20260927 00:52:29.381233 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:29.381460 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:29.381883 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [term 1 LEADER]: Becoming Leader. State: Replica: 8f9f13b0a1674c1a851d9c1080b0fa6d, State: Running, Role: LEADER
I20260927 00:52:29.382380 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38599 } }
I20260927 00:52:29.382678 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:29.384555 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38599 } } }
I20260927 00:52:29.384770 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:29.385298 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9f13b0a1674c1a851d9c1080b0fa6d. Latest consensus state: current_term: 1 leader_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9f13b0a1674c1a851d9c1080b0fa6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38599 } } }
I20260927 00:52:29.385596 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:29.388600 499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:29.388708 499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:29.388792 500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:29.389647 500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:29.397171 500 catalog_manager.cc:1347] Generated new cluster ID: 21692920595e47b5982e33a55e8d1cce
I20260927 00:52:29.397243 500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:29.399817 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:29.370937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.0.98.62:38599)
0927 00:52:29.371629 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:29.371652 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:29.372859 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 00:52:29.377687 (+ 4828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:29.377717 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:29.378227 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:29.378700 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:29.382199 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.383870 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.385881 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.386593 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.386758 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:29.391745 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:29.391751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:29.391767 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:29.391795 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:29.394072 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:29.395012 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:29.395019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:29.395026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:29.395231 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:29.395635 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:29.395639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:29.395641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:29.396081 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:29.396298 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:29.398202 (+ 1904us) server_negotiation.cc:299] Negotiation successful
0927 00:52:29.398629 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":229,"threads_started":1}
I20260927 00:52:29.423223 500 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:29.424243 500 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:29.430390 500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9f13b0a1674c1a851d9c1080b0fa6d: Generated new TSK 0
I20260927 00:52:29.431109 500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:29.463707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38599
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:29.586117 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:29.586392 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:29.586457 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:29.590583 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:29.590655 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:29.590739 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:29.596135 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38599
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:29.597499 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:29.598806 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:29.610080 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:29.610172 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:29.610085 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:29.610462 506 server_base.cc:1034] running on GCE node
I20260927 00:52:29.611029 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:29.611909 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:29.613131 506 hybrid_clock.cc:648] HybridClock initialized: now 1790470349613096 us; error 54 us; skew 500 ppm
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:29.616600 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:29.617689 506 webserver.cc:460] Webserver started at http://127.0.98.1:39623/ using document root <none> and password file <none>
I20260927 00:52:29.618278 506 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:29.618341 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:29.618517 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:29.620705 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "c46429f705d24bfd9b67979fe21774e1"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "b9a249ab1f6d78e69dd7c72fbe361153"
server_key_iv: "91556938061f8f9b84d778bb0f3b6082"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.621300 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c46429f705d24bfd9b67979fe21774e1"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "b9a249ab1f6d78e69dd7c72fbe361153"
server_key_iv: "91556938061f8f9b84d778bb0f3b6082"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.625619 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:52:29.628506 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.630131 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:29.630260 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "c46429f705d24bfd9b67979fe21774e1"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "b9a249ab1f6d78e69dd7c72fbe361153"
server_key_iv: "91556938061f8f9b84d778bb0f3b6082"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.630394 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:29.643831 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:29.647203 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:29.647387 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:29.647967 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:29.648785 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:29.648835 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.648875 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:29.648908 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.658720 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33885
I20260927 00:52:29.658800 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33885 every 8 connection(s)
I20260927 00:52:29.659734 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:29.662068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 506
I20260927 00:52:29.662187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:29.662489 392 external_mini_cluster.cc:1428] Setting key 93886381354752ccb7fded05941c3b79
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:52:29.675801 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:29.662167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55765 (local address 127.0.98.62:38599)
0927 00:52:29.663074 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:29.663090 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:29.663303 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:52:29.664765 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:29.664768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:29.664820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:29.664901 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:29.666372 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.666930 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.668072 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.668297 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.668384 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:29.671709 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:29.671713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:29.671716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:29.671745 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:29.673212 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:29.673810 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:29.673813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:29.673814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:29.673888 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:29.674318 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:29.674321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:29.674323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:29.674575 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:29.674667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:29.675413 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 00:52:29.675600 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786}
I20260927 00:52:29.676826 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:29.662506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38599 (local address 127.0.98.1:55765)
0927 00:52:29.663109 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:29.663194 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:29.664499 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:29.665121 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:29.665130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:29.665573 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:29.666210 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:29.666227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.667056 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.667058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:29.667957 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:29.667967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.668431 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.668435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:29.668590 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:29.669148 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:29.669168 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:29.671478 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:29.673346 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:29.673353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:29.673370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:29.673700 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:29.673965 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:29.673967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:29.673968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:29.674140 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:29.674671 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:29.674677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:29.675027 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 00:52:29.675303 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 00:52:29.675728 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260927 00:52:29.678494 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38599
I20260927 00:52:29.678766 635 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:29.679440 635 heartbeater.cc:511] Master 127.0.98.62:38599 requested a full tablet report, sending...
I20260927 00:52:29.681255 444 ts_manager.cc:194] Registered new tserver with Master: c46429f705d24bfd9b67979fe21774e1 (127.0.98.1:33885)
I20260927 00:52:29.682632 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:29.690575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38599
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:29.799885 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:29.800163 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:29.800207 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:29.804176 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:29.804242 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:29.804301 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:29.809537 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38599
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:29.810577 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:29.811568 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:29.821844 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:29.821844 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:29.821879 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:29.822566 640 server_base.cc:1034] running on GCE node
I20260927 00:52:29.823091 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:29.824127 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:29.825387 640 hybrid_clock.cc:648] HybridClock initialized: now 1790470349825359 us; error 41 us; skew 500 ppm
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:29.828819 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:29.829982 640 webserver.cc:460] Webserver started at http://127.0.98.2:36063/ using document root <none> and password file <none>
I20260927 00:52:29.830668 640 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:29.830735 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:29.830963 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:29.833618 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "05e8c0dedb1b4e4ba49e9bd1877e28c6"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "36c6b62c7d61795db74768e7893cfd1d"
server_key_iv: "3c9c3be610fe4605e306a75290f1ecff"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.834257 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05e8c0dedb1b4e4ba49e9bd1877e28c6"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "36c6b62c7d61795db74768e7893cfd1d"
server_key_iv: "3c9c3be610fe4605e306a75290f1ecff"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.839017 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:52:29.842314 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.844187 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:29.844367 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "05e8c0dedb1b4e4ba49e9bd1877e28c6"
format_stamp: "Formatted at 2026-09-27 00:52:29 on dist-test-slave-9fh2"
server_key: "36c6b62c7d61795db74768e7893cfd1d"
server_key_iv: "3c9c3be610fe4605e306a75290f1ecff"
server_key_version: "encryptionkey@0"
I20260927 00:52:29.844519 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:29.859423 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:29.863464 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:29.863669 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:29.864470 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:29.865484 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:29.865535 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.865578 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:29.865610 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:29.877252 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33921
I20260927 00:52:29.877326 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33921 every 8 connection(s)
I20260927 00:52:29.878438 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:29.888212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 640
I20260927 00:52:29.888362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:29.888706 392 external_mini_cluster.cc:1428] Setting key 1cec9c06574b53779d6d42cda316d737
Sep 27 00:52:29 dist-test-slave-9fh2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:29.895510 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:29.881401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52215 (local address 127.0.98.62:38599)
0927 00:52:29.881595 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:29.881601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:29.882767 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 00:52:29.884367 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:29.884371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:29.884435 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:29.884541 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:29.885882 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.886588 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.888018 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.888350 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.888479 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:29.891118 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:29.891122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:29.891125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:29.891173 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:29.892842 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:29.893451 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:29.893454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:29.893455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:29.893542 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:29.893956 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:29.893959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:29.893960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:29.894239 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:29.894346 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:29.894990 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 00:52:29.895305 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:52:29.896592 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:29.881994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38599 (local address 127.0.98.2:52215)
0927 00:52:29.882567 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:29.882621 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:29.884087 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:29.884723 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:29.884731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:29.885154 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:29.885695 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:29.885709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.886752 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.886755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:29.887860 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:29.887872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:29.888531 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:29.888535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:29.888691 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:29.889301 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:29.889319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:29.890921 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:29.892962 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:29.892969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:29.892986 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:29.893339 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:29.893637 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:29.893640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:29.893643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:29.893823 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:29.894359 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:29.894366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:29.894692 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 00:52:29.894915 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 00:52:29.895370 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260927 00:52:29.898115 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38599
I20260927 00:52:29.898505 769 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:29.899283 769 heartbeater.cc:511] Master 127.0.98.62:38599 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:29.900715 444 ts_manager.cc:194] Registered new tserver with Master: 05e8c0dedb1b4e4ba49e9bd1877e28c6 (127.0.98.2:33921)
I20260927 00:52:29.901422 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:29.919327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38599
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:30.054034 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:30.054427 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:30.054497 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:30.059026 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:30.059122 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:30.059255 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:30.065456 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38599
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:30.067059 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:30.068389 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:30.079852 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:30.079903 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:30.080343 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:30.080464 774 server_base.cc:1034] running on GCE node
I20260927 00:52:30.080994 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:30.081840 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:30.083025 774 hybrid_clock.cc:648] HybridClock initialized: now 1790470350083014 us; error 71 us; skew 500 ppm
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:30.091617 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:30.092839 774 webserver.cc:460] Webserver started at http://127.0.98.3:33231/ using document root <none> and password file <none>
I20260927 00:52:30.093551 774 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:30.093627 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:30.093885 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:30.096550 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "a0b6c49243804a9e8a2f73c80f343533"
format_stamp: "Formatted at 2026-09-27 00:52:30 on dist-test-slave-9fh2"
server_key: "582f750ff7bb4efa9e11a27a2e6c6359"
server_key_iv: "61590af950d565389d757a0dd61c5bf1"
server_key_version: "encryptionkey@0"
I20260927 00:52:30.097213 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a0b6c49243804a9e8a2f73c80f343533"
format_stamp: "Formatted at 2026-09-27 00:52:30 on dist-test-slave-9fh2"
server_key: "582f750ff7bb4efa9e11a27a2e6c6359"
server_key_iv: "61590af950d565389d757a0dd61c5bf1"
server_key_version: "encryptionkey@0"
I20260927 00:52:30.102162 774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:52:30.105616 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:30.107682 774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:30.107870 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "a0b6c49243804a9e8a2f73c80f343533"
format_stamp: "Formatted at 2026-09-27 00:52:30 on dist-test-slave-9fh2"
server_key: "582f750ff7bb4efa9e11a27a2e6c6359"
server_key_iv: "61590af950d565389d757a0dd61c5bf1"
server_key_version: "encryptionkey@0"
I20260927 00:52:30.108021 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:30.133277 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:30.137013 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:30.137220 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:30.137910 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:30.138870 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:30.138916 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:30.138976 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:30.138999 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:30.151121 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40875
I20260927 00:52:30.151206 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40875 every 8 connection(s)
I20260927 00:52:30.152305 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:30.157676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 774
I20260927 00:52:30.157822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:30.158114 392 external_mini_cluster.cc:1428] Setting key 72055f25dd9164d0b43b885004464973
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:30.168705 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:30.154638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57457 (local address 127.0.98.62:38599)
0927 00:52:30.154795 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:30.154798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:30.155904 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 00:52:30.157922 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:30.157926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:30.157994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:30.158081 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:30.159582 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.160330 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.161897 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.162133 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.162223 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:30.164815 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:30.164819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:30.164822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:30.164857 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:30.166314 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:30.166911 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:30.166914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:30.166916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:30.167005 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:30.167480 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:30.167484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:30.167485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:30.167699 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:30.167783 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:30.168386 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 00:52:30.168550 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:52:30.169581 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:30.155087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38599 (local address 127.0.98.3:57457)
0927 00:52:30.155703 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:30.155759 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:30.157631 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:30.158303 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:30.158312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:30.158749 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:30.159425 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:30.159442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.160474 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.160478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:30.161777 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:30.161788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.162292 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.162295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:30.162453 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:30.163065 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:30.163084 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:30.164618 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:30.166455 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:30.166460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:30.166475 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:30.166791 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:30.167141 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:30.167187 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:30.167189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:30.167342 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:30.167820 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:30.167826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:30.168122 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 00:52:30.168305 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:52:30.168601 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":137,"threads_started":1}
I20260927 00:52:30.171206 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38599
I20260927 00:52:30.171483 903 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:30.172082 903 heartbeater.cc:511] Master 127.0.98.62:38599 requested a full tablet report, sending...
I20260927 00:52:30.173600 443 ts_manager.cc:194] Registered new tserver with Master: a0b6c49243804a9e8a2f73c80f343533 (127.0.98.3:40875)
I20260927 00:52:30.174301 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57457
I20260927 00:52:30.186482 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:30.217597 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:30.206336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.1:33885)
0927 00:52:30.206693 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:30.206702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:30.206858 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:52:30.206999 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:30.207005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:30.207265 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:30.207420 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:30.208292 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.209122 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.210242 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.210676 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.210791 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:30.213727 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:30.213738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:30.213752 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:30.213781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:30.215818 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:30.216273 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:30.216278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:30.216280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:30.216363 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:30.216604 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:30.216608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:30.216609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:30.216863 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:30.217077 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:30.217278 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:52:30.217425 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
W20260927 00:52:30.219023 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53076
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:30.248427 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:30.239279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.0.98.1:33885)
0927 00:52:30.239466 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:30.239471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:30.239619 (+ 148us) server_negotiation.cc:407] Connection header received
0927 00:52:30.239783 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:30.239786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:30.239832 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:30.239907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:30.240686 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.241201 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.242043 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:30.242261 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:30.242349 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:30.245221 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:30.245224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:30.245226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:30.245252 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:30.246670 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:30.247179 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:30.247183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:30.247185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:30.247281 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:30.247590 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:30.247593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:30.247594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:30.247791 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:30.247943 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:30.248151 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:52:30.248270 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:52:30.249871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 506
I20260927 00:52:30.257004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 640
I20260927 00:52:30.265501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 774
I20260927 00:52:30.272068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 412
2026-09-27T00:52:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2252 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[917](info): set up 2 sockets
Sep 27 00:52:30 dist-test-slave-9fh2 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 00:52:31.242991 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:31 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:31.374716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:31.501992 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:31.502352 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:31.502424 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:31.506883 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:31.506985 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:31.507014 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:31.507035 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:31.507054 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:31.513238 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:31.514657 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:31.516253 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:31.526738 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:31.526748 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:31.526952 930 server_base.cc:1034] running on GCE node
W20260927 00:52:31.526746 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:31.527999 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:31.529386 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:31.530579 930 hybrid_clock.cc:648] HybridClock initialized: now 1790470351530545 us; error 55 us; skew 500 ppm
Sep 27 00:52:31 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:31.534039 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:31.535403 930 webserver.cc:460] Webserver started at http://127.0.98.62:45987/ using document root <none> and password file <none>
I20260927 00:52:31.536093 930 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:31.536173 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:31.536453 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:31.539067 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "842dba2133db41eda1bb4f1b5857db09"
format_stamp: "Formatted at 2026-09-27 00:52:31 on dist-test-slave-9fh2"
server_key: "72a908b8f4ebbc3e7699f7d8b5a41a50"
server_key_iv: "8999d21d69e33adf0480b0625d2b511c"
server_key_version: "encryptionkey@0"
I20260927 00:52:31.539870 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "842dba2133db41eda1bb4f1b5857db09"
format_stamp: "Formatted at 2026-09-27 00:52:31 on dist-test-slave-9fh2"
server_key: "72a908b8f4ebbc3e7699f7d8b5a41a50"
server_key_iv: "8999d21d69e33adf0480b0625d2b511c"
server_key_version: "encryptionkey@0"
I20260927 00:52:31.545377 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:52:31.549885 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:31.552254 930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:52:31.552464 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "842dba2133db41eda1bb4f1b5857db09"
format_stamp: "Formatted at 2026-09-27 00:52:31 on dist-test-slave-9fh2"
server_key: "72a908b8f4ebbc3e7699f7d8b5a41a50"
server_key_iv: "8999d21d69e33adf0480b0625d2b511c"
server_key_version: "encryptionkey@0"
I20260927 00:52:31.552641 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:31.571852 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:31.575533 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:31.575730 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:31.583019 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38457
I20260927 00:52:31.583078 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38457 every 8 connection(s)
I20260927 00:52:31.584136 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:31.587805 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:31.592705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 930
I20260927 00:52:31.592821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:31.593098 392 external_mini_cluster.cc:1428] Setting key 58832292dec196145cb3ddf29f8e307a
I20260927 00:52:31.595211 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09: Bootstrap starting.
I20260927 00:52:31.598582 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:31.599695 998 log.cc:824] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:31.602331 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09: No bootstrap required, opened a new log
Sep 27 00:52:31 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:31.606952 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842dba2133db41eda1bb4f1b5857db09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38457 } }
I20260927 00:52:31.607326 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:31.607383 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842dba2133db41eda1bb4f1b5857db09, State: Initialized, Role: FOLLOWER
I20260927 00:52:31.607965 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842dba2133db41eda1bb4f1b5857db09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38457 } }
I20260927 00:52:31.608116 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:31.608192 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:31.608302 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:31.609457 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842dba2133db41eda1bb4f1b5857db09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38457 } }
I20260927 00:52:31.609867 998 leader_election.cc:302] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842dba2133db41eda1bb4f1b5857db09; no voters:
I20260927 00:52:31.610240 998 leader_election.cc:288] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:31.610539 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:31.610735 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [term 1 LEADER]: Becoming Leader. State: Replica: 842dba2133db41eda1bb4f1b5857db09, State: Running, Role: LEADER
I20260927 00:52:31.611102 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842dba2133db41eda1bb4f1b5857db09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38457 } }
I20260927 00:52:31.611680 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:31.594881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.0.98.62:38457)
0927 00:52:31.595540 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:31.595561 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:31.595614 (+ 53us) server_negotiation.cc:407] Connection header received
0927 00:52:31.596592 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:31.596623 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:31.597162 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:31.597613 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:31.598338 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:31.599971 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:31.600890 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:31.601606 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:31.601787 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:31.604517 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:31.604528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:31.604565 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:31.604617 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:31.608099 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:31.608616 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:31.608623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:31.608628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:31.608755 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:31.609016 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:31.609020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:31.609021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:31.609500 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:31.609691 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:31.610125 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 00:52:31.610499 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":185,"threads_started":1}
I20260927 00:52:31.611907 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:31.613107 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842dba2133db41eda1bb4f1b5857db09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842dba2133db41eda1bb4f1b5857db09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38457 } } }
I20260927 00:52:31.613256 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:31.613631 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842dba2133db41eda1bb4f1b5857db09. Latest consensus state: current_term: 1 leader_uuid: "842dba2133db41eda1bb4f1b5857db09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842dba2133db41eda1bb4f1b5857db09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38457 } } }
I20260927 00:52:31.613762 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:31.613853 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:31.617548 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:31.624845 1012 catalog_manager.cc:1347] Generated new cluster ID: 523c674c8ac5485e86ebf077843146ab
I20260927 00:52:31.624944 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:31.653554 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:31.654930 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:31.672472 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842dba2133db41eda1bb4f1b5857db09: Generated new TSK 0
I20260927 00:52:31.673408 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:31.710604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38457
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:31.840418 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:31.840785 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:31.840854 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:31.845384 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:31.845480 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:31.845573 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:31.851760 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:31.853217 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:31.854501 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:31.865943 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:31.866204 1024 server_base.cc:1034] running on GCE node
W20260927 00:52:31.865933 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:31.865962 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:31.866880 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:31.867743 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:31.868938 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790470351868916 us; error 53 us; skew 500 ppm
Sep 27 00:52:31 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:31.871770 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:31.872720 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35321/ using document root <none> and password file <none>
I20260927 00:52:31.873338 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:31.873397 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:31.873579 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:31.875830 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "acd4034facbf45de9f4c65069e671a89"
format_stamp: "Formatted at 2026-09-27 00:52:31 on dist-test-slave-9fh2"
server_key: "834cdcdd4b136839e82d09fbf5fbbbc9"
server_key_iv: "d0de8558d72a13f4a64c40917a91edde"
server_key_version: "encryptionkey@0"
I20260927 00:52:31.876360 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "acd4034facbf45de9f4c65069e671a89"
format_stamp: "Formatted at 2026-09-27 00:52:31 on dist-test-slave-9fh2"
server_key: "834cdcdd4b136839e82d09fbf5fbbbc9"
server_key_iv: "d0de8558d72a13f4a64c40917a91edde"
server_key_version: "encryptionkey@0"
I20260927 00:52:31.880640 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:31.883417 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:31.884938 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:31.885111 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "acd4034facbf45de9f4c65069e671a89"
format_stamp: "Formatted at 2026-09-27 00:52:31 on dist-test-slave-9fh2"
server_key: "834cdcdd4b136839e82d09fbf5fbbbc9"
server_key_iv: "d0de8558d72a13f4a64c40917a91edde"
server_key_version: "encryptionkey@0"
I20260927 00:52:31.885233 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:31.917831 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:31.921067 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:31.921227 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:31.921850 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:31.922789 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:31.922852 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:31.922906 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:31.922941 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:31.932530 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37257
I20260927 00:52:31.932574 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37257 every 8 connection(s)
I20260927 00:52:31.933558 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:31.938687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1024
I20260927 00:52:31.938799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:31.939067 392 external_mini_cluster.cc:1428] Setting key a966f6f761394213c20723d1dfd191e3
Sep 27 00:52:31 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:31.949227 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:31.935982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50485 (local address 127.0.98.62:38457)
0927 00:52:31.936159 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:31.936165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:31.937201 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 00:52:31.938768 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:31.938772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:31.938837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:31.938951 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:31.940851 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:31.941407 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:31.942734 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:31.942961 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:31.943061 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:31.945576 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:31.945579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:31.945581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:31.945608 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:31.947192 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:31.947693 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:31.947696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:31.947698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:31.947771 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:31.948114 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:31.948117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:31.948118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:31.948301 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:31.948384 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:31.948917 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 00:52:31.949059 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:52:31.950156 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:31.936398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38457 (local address 127.0.98.1:50485)
0927 00:52:31.936998 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:31.937058 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:31.938524 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:31.939180 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:31.939191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:31.939747 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:31.940642 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:31.940670 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:31.941571 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:31.941574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:31.942585 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:31.942609 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:31.943112 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:31.943117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:31.943327 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:31.943911 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:31.943928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:31.945385 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:31.947298 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:31.947305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:31.947323 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:31.947599 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:31.947849 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:31.947852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:31.947854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:31.947991 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:31.948397 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:31.948405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:31.948685 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 00:52:31.948866 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:52:31.949219 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":120,"threads_started":1}
I20260927 00:52:31.951699 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38457
I20260927 00:52:31.952096 1153 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:52:31.952890 1153 heartbeater.cc:511] Master 127.0.98.62:38457 requested a full tablet report, sending...
I20260927 00:52:31.954957 959 ts_manager.cc:194] Registered new tserver with Master: acd4034facbf45de9f4c65069e671a89 (127.0.98.1:37257)
I20260927 00:52:31.956231 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:31.970875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38457
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:32.101191 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:32.101544 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:32.101598 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:32.106264 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:32.106366 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:32.106460 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:32.112926 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:32.114439 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:32.115836 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:32.127378 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:32.127483 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:32.127384 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:32.127650 1158 server_base.cc:1034] running on GCE node
I20260927 00:52:32.128270 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:32.129124 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:32.130317 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790470352130290 us; error 48 us; skew 500 ppm
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:32.133734 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:32.134895 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38965/ using document root <none> and password file <none>
I20260927 00:52:32.135636 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:32.135711 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:32.135950 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:32.138463 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "07e0d0f90d4e4b1090b6c781caaae85c"
format_stamp: "Formatted at 2026-09-27 00:52:32 on dist-test-slave-9fh2"
server_key: "7b0868ef9682dae25164aa2e34dbde4b"
server_key_iv: "256213865c06660140e159ca2b46e72a"
server_key_version: "encryptionkey@0"
I20260927 00:52:32.139067 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "07e0d0f90d4e4b1090b6c781caaae85c"
format_stamp: "Formatted at 2026-09-27 00:52:32 on dist-test-slave-9fh2"
server_key: "7b0868ef9682dae25164aa2e34dbde4b"
server_key_iv: "256213865c06660140e159ca2b46e72a"
server_key_version: "encryptionkey@0"
I20260927 00:52:32.143718 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:32.146715 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:32.148458 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:32.148603 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "07e0d0f90d4e4b1090b6c781caaae85c"
format_stamp: "Formatted at 2026-09-27 00:52:32 on dist-test-slave-9fh2"
server_key: "7b0868ef9682dae25164aa2e34dbde4b"
server_key_iv: "256213865c06660140e159ca2b46e72a"
server_key_version: "encryptionkey@0"
I20260927 00:52:32.148752 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:32.178256 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:32.181756 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:32.181929 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:32.182552 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:32.183465 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:32.183516 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:32.183558 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:32.183590 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:32.193338 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35599
I20260927 00:52:32.193428 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35599 every 8 connection(s)
I20260927 00:52:32.194317 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:32.198791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1158
I20260927 00:52:32.198894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:32.199110 392 external_mini_cluster.cc:1428] Setting key 512242c5bca8f0c87b4e80041ef1f461
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:32.212754 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.196824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57537 (local address 127.0.98.62:38457)
0927 00:52:32.197011 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.197015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.197985 (+ 970us) server_negotiation.cc:407] Connection header received
0927 00:52:32.199691 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.199695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.199760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.199887 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:32.201808 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.202554 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.204628 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.204850 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.204938 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.208491 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:32.208494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:32.208496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:32.208524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:32.210310 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.210985 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.210988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.210990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.211068 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.211471 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.211474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.211476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.211684 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:32.211829 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.212416 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.212589 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:52:32.213835 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.197221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38457 (local address 127.0.98.2:57537)
0927 00:52:32.197767 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.197845 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.199390 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.200296 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.200318 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.200942 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:32.201619 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.201638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.202808 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.202813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.204458 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.204476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.205040 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.205045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.205301 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.206117 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:32.206145 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:32.208284 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:32.210470 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:32.210479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:32.210498 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:32.210879 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:32.211188 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:32.211191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:32.211193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:32.211351 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:32.211839 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:32.211846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:32.212166 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.212346 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.212789 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
I20260927 00:52:32.215855 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38457
I20260927 00:52:32.216173 1287 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:32.216738 1287 heartbeater.cc:511] Master 127.0.98.62:38457 requested a full tablet report, sending...
I20260927 00:52:32.218268 959 ts_manager.cc:194] Registered new tserver with Master: 07e0d0f90d4e4b1090b6c781caaae85c (127.0.98.2:35599)
I20260927 00:52:32.219303 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:32.230947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38457
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:32.362511 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:32.362905 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:32.362957 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:32.367669 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:32.367781 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:32.367856 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:32.374190 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38457
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:32.375828 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:32.377130 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:32.388906 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:32.388911 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:32.388906 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:32.388994 1292 server_base.cc:1034] running on GCE node
I20260927 00:52:32.389866 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:32.390635 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:32.391839 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790470352391777 us; error 78 us; skew 500 ppm
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:32.395052 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:32.396291 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34557/ using document root <none> and password file <none>
I20260927 00:52:32.396986 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:32.397058 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:32.397287 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:32.399828 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9be16ffaa3149308e55ee417f9ccac9"
format_stamp: "Formatted at 2026-09-27 00:52:32 on dist-test-slave-9fh2"
server_key: "d058705d4dd79aecd29771c8d0e0c1d1"
server_key_iv: "df69aa910a4785c19c68cdaf4eb815a5"
server_key_version: "encryptionkey@0"
I20260927 00:52:32.400454 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9be16ffaa3149308e55ee417f9ccac9"
format_stamp: "Formatted at 2026-09-27 00:52:32 on dist-test-slave-9fh2"
server_key: "d058705d4dd79aecd29771c8d0e0c1d1"
server_key_iv: "df69aa910a4785c19c68cdaf4eb815a5"
server_key_version: "encryptionkey@0"
I20260927 00:52:32.405054 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:52:32.408092 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:32.409574 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:32.409696 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "e9be16ffaa3149308e55ee417f9ccac9"
format_stamp: "Formatted at 2026-09-27 00:52:32 on dist-test-slave-9fh2"
server_key: "d058705d4dd79aecd29771c8d0e0c1d1"
server_key_iv: "df69aa910a4785c19c68cdaf4eb815a5"
server_key_version: "encryptionkey@0"
I20260927 00:52:32.409818 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:32.432178 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:32.435967 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:32.436152 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:32.436810 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:32.437772 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:32.437830 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:32.437878 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:32.437893 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:32.448693 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40839
I20260927 00:52:32.448789 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40839 every 8 connection(s)
I20260927 00:52:32.449798 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:32.458844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1292
I20260927 00:52:32.458974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:32.459259 392 external_mini_cluster.cc:1428] Setting key fa725a7767fdb0c6f8bd5be2facaebfb
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:32.465579 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.452188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34015 (local address 127.0.98.62:38457)
0927 00:52:32.452413 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.452417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.453632 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 00:52:32.455127 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.455130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.455211 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.455302 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:32.456653 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.457194 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.458390 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.458633 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.458740 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.461823 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:32.461826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:32.461829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:32.461856 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:32.463551 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.464079 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.464083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.464084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.464155 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.464482 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.464485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.464486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.464684 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:32.464766 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.465292 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.465408 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 00:52:32.466522 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.452762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38457 (local address 127.0.98.3:34015)
0927 00:52:32.453453 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.453517 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.454888 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.455510 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.455518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.455914 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:32.456482 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.456522 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.457311 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.457314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.458279 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.458289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.458842 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.458847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.459092 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.459853 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:32.459877 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:32.461666 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:32.463668 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:32.463676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:32.463694 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:32.463986 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:32.464236 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:32.464239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:32.464241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:32.464385 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:32.464780 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:32.464788 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:32.465065 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.465242 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.465603 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":139,"threads_started":1}
I20260927 00:52:32.468000 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38457
I20260927 00:52:32.468245 1421 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:32.468792 1421 heartbeater.cc:511] Master 127.0.98.62:38457 requested a full tablet report, sending...
I20260927 00:52:32.469918 959 ts_manager.cc:194] Registered new tserver with Master: e9be16ffaa3149308e55ee417f9ccac9 (127.0.98.3:40839)
I20260927 00:52:32.470507 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34015
I20260927 00:52:32.473412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:52:32.485168 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.477782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.98.62:38457)
0927 00:52:32.477933 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.477938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.478126 (+ 188us) server_negotiation.cc:407] Connection header received
0927 00:52:32.478267 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.478270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.478313 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.478378 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:32.479100 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.479659 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.480470 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.480677 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.480760 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.482148 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:32.482158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:32.482160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:32.482182 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:32.483553 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.483943 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.483947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.483949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.484020 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.484256 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.484259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.484261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.484484 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:32.484604 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.484840 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.484946 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:52:32.491539 959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:32.494807 959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:32.512426 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.505766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.0.98.1:37257)
0927 00:52:32.507037 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.507046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.507075 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:52:32.507206 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.507213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.507492 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.507677 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:32.508009 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.509284 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.510889 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.511641 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.511752 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.511935 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.512107 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.512225 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":130,"threads_started":1}
I20260927 00:52:32.512418 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.505560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37257 (local address 127.0.0.1:39658)
0927 00:52:32.506459 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.506510 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.506650 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.507670 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.507676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.507701 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:32.507847 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.507854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.509417 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.509420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.510742 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.510752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.511781 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.511784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.511902 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.511945 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.512064 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.512243 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":139,"thread_start_us":86,"threads_started":1}
I20260927 00:52:32.512609 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.506545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.98.3:40839)
0927 00:52:32.507363 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.507374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.507397 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:52:32.507473 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.507478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.507661 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.507821 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:32.508152 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.508969 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.510876 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.511798 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.511891 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.512005 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.512325 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.512430 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":151,"threads_started":1}
I20260927 00:52:32.513279 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.505726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40839 (local address 127.0.0.1:37754)
0927 00:52:32.506867 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.506891 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.506971 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.507894 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.507898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.507918 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:32.508044 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.508050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.509097 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.509100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.510720 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.510728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.512052 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.512055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.512170 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.512189 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.512243 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.512489 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":328,"spinlock_wait_cycles":4352,"thread_start_us":144,"threads_started":1}
I20260927 00:52:32.513484 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.506272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35599 (local address 127.0.0.1:39362)
0927 00:52:32.507139 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.507175 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.507251 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.507936 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.507939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.507972 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:32.508091 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.508098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.509465 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.509468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.510628 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.510640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.512010 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.512014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.512139 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.512169 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.512241 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.512347 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":60,"spinlock_wait_cycles":896,"thread_start_us":118,"threads_started":1}
I20260927 00:52:32.513948 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.506463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.98.2:35599)
0927 00:52:32.506922 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.506937 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.507335 (+ 398us) server_negotiation.cc:407] Connection header received
0927 00:52:32.507421 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.507427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.507639 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.507794 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:32.508214 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.509335 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.510759 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.511647 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.511777 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.511946 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.513713 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.513812 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":168,"threads_started":1}
I20260927 00:52:32.516438 1087 tablet_service.cc:1467] Processing CreateTablet for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae (DEFAULT_TABLE table=test-table [id=093e4a56bf7142f3b4816529ccc7e050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:32.517095 1221 tablet_service.cc:1467] Processing CreateTablet for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae (DEFAULT_TABLE table=test-table [id=093e4a56bf7142f3b4816529ccc7e050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:32.517094 1355 tablet_service.cc:1467] Processing CreateTablet for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae (DEFAULT_TABLE table=test-table [id=093e4a56bf7142f3b4816529ccc7e050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:32.517652 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:32.518388 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:32.518388 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:32.525264 1438 tablet_bootstrap.cc:492] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89: Bootstrap starting.
I20260927 00:52:32.526988 1439 tablet_bootstrap.cc:492] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c: Bootstrap starting.
I20260927 00:52:32.528103 1440 tablet_bootstrap.cc:492] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9: Bootstrap starting.
I20260927 00:52:32.528272 1438 tablet_bootstrap.cc:654] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:32.529409 1439 tablet_bootstrap.cc:654] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:32.529899 1438 log.cc:824] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:32.530596 1440 tablet_bootstrap.cc:654] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:32.530777 1439 log.cc:824] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:32.532045 1440 log.cc:824] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:32.532642 1438 tablet_bootstrap.cc:492] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89: No bootstrap required, opened a new log
I20260927 00:52:32.532898 1438 ts_tablet_manager.cc:1397] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:52:32.533043 1439 tablet_bootstrap.cc:492] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c: No bootstrap required, opened a new log
I20260927 00:52:32.533242 1439 ts_tablet_manager.cc:1397] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 00:52:32.534077 1440 tablet_bootstrap.cc:492] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9: No bootstrap required, opened a new log
I20260927 00:52:32.534289 1440 ts_tablet_manager.cc:1397] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:52:32.537657 1439 raft_consensus.cc:348] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.537802 1440 raft_consensus.cc:348] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.538077 1439 raft_consensus.cc:374] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:32.538080 1440 raft_consensus.cc:374] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:32.538154 1440 raft_consensus.cc:729] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9be16ffaa3149308e55ee417f9ccac9, State: Initialized, Role: FOLLOWER
I20260927 00:52:32.538162 1439 raft_consensus.cc:729] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e0d0f90d4e4b1090b6c781caaae85c, State: Initialized, Role: FOLLOWER
I20260927 00:52:32.538638 1440 consensus_queue.cc:260] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.538795 1439 consensus_queue.cc:260] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.539474 1421 heartbeater.cc:503] Master 127.0.98.62:38457 was elected leader, sending a full tablet report...
I20260927 00:52:32.539179 1438 raft_consensus.cc:348] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.539601 1438 raft_consensus.cc:374] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:32.539677 1438 raft_consensus.cc:729] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd4034facbf45de9f4c65069e671a89, State: Initialized, Role: FOLLOWER
I20260927 00:52:32.540262 1439 ts_tablet_manager.cc:1428] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260927 00:52:32.540258 1438 consensus_queue.cc:260] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.540763 1287 heartbeater.cc:503] Master 127.0.98.62:38457 was elected leader, sending a full tablet report...
I20260927 00:52:32.541447 1438 ts_tablet_manager.cc:1428] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260927 00:52:32.541539 1440 ts_tablet_manager.cc:1428] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:52:32.541828 1153 heartbeater.cc:503] Master 127.0.98.62:38457 was elected leader, sending a full tablet report...
I20260927 00:52:32.543943 1445 raft_consensus.cc:482] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:32.544579 1445 raft_consensus.cc:504] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.546985 1445 leader_election.cc:288] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd4034facbf45de9f4c65069e671a89 (127.0.98.1:37257), e9be16ffaa3149308e55ee417f9ccac9 (127.0.98.3:40839)
I20260927 00:52:32.550052 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.546281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37257 (local address 127.0.98.2:42089)
0927 00:52:32.546414 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.546430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.546511 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.546859 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.546863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.546885 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:32.547027 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.547036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.548038 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.548040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.548895 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.548903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.549589 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.549593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.549706 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.549730 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.549796 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.549889 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:52:32.550052 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.546465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42089 (local address 127.0.98.1:37257)
0927 00:52:32.546608 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.546613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.546629 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:52:32.546688 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.546691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.546743 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.546823 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:32.547214 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.547960 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.549015 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.549477 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.549576 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.549651 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.549818 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.549898 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:52:32.550793 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.546789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40501 (local address 127.0.98.3:40839)
0927 00:52:32.546958 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.546961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.547522 (+ 561us) server_negotiation.cc:407] Connection header received
0927 00:52:32.547599 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.547601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.547647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.547726 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:32.548041 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.548552 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.549683 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.550104 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.550203 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.550280 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.550589 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.550659 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 00:52:32.550805 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.546665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40839 (local address 127.0.98.2:40501)
0927 00:52:32.547375 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:32.547391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:32.547515 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:32.547744 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:32.547747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:32.547760 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:32.547887 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.547893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.548689 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.548693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.549562 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:32.549573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.550361 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.550365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:32.550465 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.550489 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:52:32.550561 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:52:32.550653 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":120,"threads_started":1}
I20260927 00:52:32.550796 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4cadeed2b4ef8a6e79986a7c8d5ae" candidate_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd4034facbf45de9f4c65069e671a89" is_pre_election: true
I20260927 00:52:32.551223 1107 raft_consensus.cc:2454] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e0d0f90d4e4b1090b6c781caaae85c in term 0.
I20260927 00:52:32.551512 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4cadeed2b4ef8a6e79986a7c8d5ae" candidate_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9be16ffaa3149308e55ee417f9ccac9" is_pre_election: true
I20260927 00:52:32.551851 1375 raft_consensus.cc:2454] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e0d0f90d4e4b1090b6c781caaae85c in term 0.
I20260927 00:52:32.551823 1175 leader_election.cc:302] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e0d0f90d4e4b1090b6c781caaae85c, acd4034facbf45de9f4c65069e671a89; no voters:
I20260927 00:52:32.552145 1445 raft_consensus.cc:2781] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:32.552235 1445 raft_consensus.cc:482] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:32.552279 1445 raft_consensus.cc:3037] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:32.553608 1445 raft_consensus.cc:504] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.554039 1445 leader_election.cc:288] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [CANDIDATE]: Term 1 election: Requested vote from peers acd4034facbf45de9f4c65069e671a89 (127.0.98.1:37257), e9be16ffaa3149308e55ee417f9ccac9 (127.0.98.3:40839)
I20260927 00:52:32.554414 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4cadeed2b4ef8a6e79986a7c8d5ae" candidate_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd4034facbf45de9f4c65069e671a89"
I20260927 00:52:32.554483 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4cadeed2b4ef8a6e79986a7c8d5ae" candidate_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9be16ffaa3149308e55ee417f9ccac9"
I20260927 00:52:32.554561 1107 raft_consensus.cc:3037] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:32.554654 1375 raft_consensus.cc:3037] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:32.555960 1107 raft_consensus.cc:2454] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P acd4034facbf45de9f4c65069e671a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e0d0f90d4e4b1090b6c781caaae85c in term 1.
I20260927 00:52:32.556133 1375 raft_consensus.cc:2454] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P e9be16ffaa3149308e55ee417f9ccac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e0d0f90d4e4b1090b6c781caaae85c in term 1.
I20260927 00:52:32.556314 1175 leader_election.cc:302] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e0d0f90d4e4b1090b6c781caaae85c, acd4034facbf45de9f4c65069e671a89; no voters:
I20260927 00:52:32.556535 1445 raft_consensus.cc:2781] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:32.556804 1445 raft_consensus.cc:686] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [term 1 LEADER]: Becoming Leader. State: Replica: 07e0d0f90d4e4b1090b6c781caaae85c, State: Running, Role: LEADER
I20260927 00:52:32.557257 1445 consensus_queue.cc:237] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } }
I20260927 00:52:32.560915 962 catalog_manager.cc:5495] T e8a4cadeed2b4ef8a6e79986a7c8d5ae P 07e0d0f90d4e4b1090b6c781caaae85c reported cstate change: term changed from 0 to 1, leader changed from <none> to 07e0d0f90d4e4b1090b6c781caaae85c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd4034facbf45de9f4c65069e671a89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9be16ffaa3149308e55ee417f9ccac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07e0d0f90d4e4b1090b6c781caaae85c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35599 } health_report { overall_health: HEALTHY } } }
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:32.580302 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.570452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.0.98.1:37257)
0927 00:52:32.570581 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.570585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.570789 (+ 204us) server_negotiation.cc:407] Connection header received
0927 00:52:32.570916 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.570918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.570969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.571057 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:32.571942 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.572487 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.573538 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.573766 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.573861 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.576657 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:32.576668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:32.576685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:32.576712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:32.578387 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.578882 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.578887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.578888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.578970 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.579329 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.579332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.579333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.579625 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:32.579795 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.580054 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.580163 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:32.616727 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.604939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.0.98.1:37257)
0927 00:52:32.605157 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.605163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.605375 (+ 212us) server_negotiation.cc:407] Connection header received
0927 00:52:32.605606 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.605609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.605663 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.605746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:32.606727 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.607390 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.608731 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.609075 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.609201 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.612927 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:32.612933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:32.612937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:32.612990 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:32.614823 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.615457 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.615460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.615461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.615552 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.615801 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.615803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.615804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.616000 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:32.616107 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.616445 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.616575 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:32.644899 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:32.635632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.98.1:37257)
0927 00:52:32.635835 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:32.635840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:32.635999 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:52:32.636171 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:32.636175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:32.636226 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:32.636301 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:32.637069 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.637659 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.638517 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:32.638737 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:32.638824 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:32.641747 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:32.641753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:32.641756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:32.641798 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:32.643115 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.643633 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.643636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.643637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.643712 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:32.643924 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:32.643926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:32.643927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:32.644113 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:32.644217 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:32.644591 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 00:52:32.644717 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 00:52:32.646602 1067 tablet_service.cc:2908] Rejecting scan request for tablet e8a4cadeed2b4ef8a6e79986a7c8d5ae: Uninitialized: clean time has not yet been initialized
I20260927 00:52:32.649001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1024
I20260927 00:52:32.655957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1158
I20260927 00:52:32.662075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1292
I20260927 00:52:32.668205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 930
2026-09-27T00:52:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2394 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[1462](info): set up 2 sockets
Sep 27 00:52:32 dist-test-slave-9fh2 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 00:52:33.631434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:33 dist-test-slave-9fh2 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:33.762259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:33.878353 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:33.878638 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:33.878679 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:33.882673 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:33.882745 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:52:33.882761 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:33.882778 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:33.882793 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:33.882807 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:33.888263 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37667
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:33.889396 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:33.890420 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:33.899299 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:33.899320 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:33.899317 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:33.900032 1475 server_base.cc:1034] running on GCE node
I20260927 00:52:33.900766 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:33.902276 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:33.903455 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790470353903436 us; error 40 us; skew 500 ppm
Sep 27 00:52:33 dist-test-slave-9fh2 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:33.906543 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:33.907685 1475 webserver.cc:460] Webserver started at http://127.0.98.62:38567/ using document root <none> and password file <none>
I20260927 00:52:33.908381 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:33.908452 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:33.908684 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:33.911223 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "f0482dc7f9e44f3196ddf00731a440e0"
format_stamp: "Formatted at 2026-09-27 00:52:33 on dist-test-slave-9fh2"
server_key: "a1b2120f1b862af9b63c6af6350f4e2f"
server_key_iv: "bb5190bc609f179480f09d1a810e8b0a"
server_key_version: "encryptionkey@0"
I20260927 00:52:33.911846 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0482dc7f9e44f3196ddf00731a440e0"
format_stamp: "Formatted at 2026-09-27 00:52:33 on dist-test-slave-9fh2"
server_key: "a1b2120f1b862af9b63c6af6350f4e2f"
server_key_iv: "bb5190bc609f179480f09d1a810e8b0a"
server_key_version: "encryptionkey@0"
I20260927 00:52:33.916566 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:33.920229 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:33.922133 1475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:33.922304 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "f0482dc7f9e44f3196ddf00731a440e0"
format_stamp: "Formatted at 2026-09-27 00:52:33 on dist-test-slave-9fh2"
server_key: "a1b2120f1b862af9b63c6af6350f4e2f"
server_key_iv: "bb5190bc609f179480f09d1a810e8b0a"
server_key_version: "encryptionkey@0"
I20260927 00:52:33.922451 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:33.947810 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:33.951956 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:33.952167 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:33.961978 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37667
I20260927 00:52:33.962005 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37667 every 8 connection(s)
I20260927 00:52:33.963332 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:33.966563 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:33.969642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1475
I20260927 00:52:33.969801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:33.970121 392 external_mini_cluster.cc:1428] Setting key 8b98382531ac00d39c1640dc1f256405
I20260927 00:52:33.972838 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: Bootstrap starting.
I20260927 00:52:33.975333 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:33.976154 1553 log.cc:824] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:33.978220 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: No bootstrap required, opened a new log
Sep 27 00:52:33 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:33.982267 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0482dc7f9e44f3196ddf00731a440e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260927 00:52:33.982568 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:33.982649 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0482dc7f9e44f3196ddf00731a440e0, State: Initialized, Role: FOLLOWER
I20260927 00:52:33.983088 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0482dc7f9e44f3196ddf00731a440e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260927 00:52:33.983255 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:33.983325 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:33.983423 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:33.984355 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0482dc7f9e44f3196ddf00731a440e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260927 00:52:33.984695 1553 leader_election.cc:302] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0482dc7f9e44f3196ddf00731a440e0; no voters:
I20260927 00:52:33.985018 1553 leader_election.cc:288] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:33.985105 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:33.985298 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [term 1 LEADER]: Becoming Leader. State: Replica: f0482dc7f9e44f3196ddf00731a440e0, State: Running, Role: LEADER
I20260927 00:52:33.985610 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0482dc7f9e44f3196ddf00731a440e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260927 00:52:33.986403 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:33.987988 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:33.971961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.98.62:37667)
0927 00:52:33.972615 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:33.972636 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:33.972694 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:52:33.973645 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:33.973670 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:33.974158 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:33.974592 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:33.975498 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:33.976668 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:33.977576 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:33.978134 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:33.978313 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:33.981365 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:33.981371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:33.981394 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:33.981432 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:33.983730 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:33.984263 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:33.984269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:33.984275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:33.984423 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:33.984745 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:33.984748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:33.984749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:33.985439 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:33.985710 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:33.986380 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 00:52:33.986728 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":179,"threads_started":1}
I20260927 00:52:33.988729 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0482dc7f9e44f3196ddf00731a440e0. Latest consensus state: current_term: 1 leader_uuid: "f0482dc7f9e44f3196ddf00731a440e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0482dc7f9e44f3196ddf00731a440e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } } }
I20260927 00:52:33.988518 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0482dc7f9e44f3196ddf00731a440e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0482dc7f9e44f3196ddf00731a440e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } } }
I20260927 00:52:33.988842 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:33.988842 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0 [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:33.991389 1572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:33.991472 1572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:33.991576 1573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:33.992306 1573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:33.998986 1573 catalog_manager.cc:1347] Generated new cluster ID: 2decdbe2a6ea4edcb4891d259bfec06a
I20260927 00:52:33.999075 1573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:34.011126 1573 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:34.012868 1573 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:34.021435 1573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: Generated new TSK 0
I20260927 00:52:34.022593 1573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:34.063757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:34.176841 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:34.177151 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:34.177211 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:34.181288 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:34.181363 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:34.181416 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:52:34.181463 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:34.186894 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:34.188334 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:34.189489 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:34.200354 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:34.200500 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:34.200354 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:34.200524 1579 server_base.cc:1034] running on GCE node
I20260927 00:52:34.201220 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:34.202028 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:34.203255 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790470354203227 us; error 46 us; skew 500 ppm
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:34.206773 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:34.207993 1579 webserver.cc:460] Webserver started at http://127.0.98.1:36475/ using document root <none> and password file <none>
I20260927 00:52:34.208796 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:34.208879 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:34.209131 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:34.212240 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "dcb871022e0547d18e6cf5636a8eaeca"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "0adc9b5b31fcba608cd5f55a220252da"
server_key_iv: "30674a3c756780e991397ae6ae40a347"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.213053 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dcb871022e0547d18e6cf5636a8eaeca"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "0adc9b5b31fcba608cd5f55a220252da"
server_key_iv: "30674a3c756780e991397ae6ae40a347"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.218546 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:52:34.222235 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.224061 1579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:34.224232 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "dcb871022e0547d18e6cf5636a8eaeca"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "0adc9b5b31fcba608cd5f55a220252da"
server_key_iv: "30674a3c756780e991397ae6ae40a347"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.224378 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:34.245268 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:34.249446 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:34.249694 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:34.251891 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:34.251963 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.252027 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:34.252056 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:34.264664 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36889
I20260927 00:52:34.264739 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36889 every 8 connection(s)
I20260927 00:52:34.265875 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:34.269924 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.252843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49039 (local address 127.0.98.62:37667)
0927 00:52:34.253056 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.253063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.254351 (+ 1288us) server_negotiation.cc:407] Connection header received
0927 00:52:34.256072 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.256077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.256156 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.256288 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.258037 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.258840 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.261076 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.261348 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.261439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.265227 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.265232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.265234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.265262 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.267065 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.267709 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.267712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.267713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.267808 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.268275 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.268278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.268280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.268668 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.268774 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.269606 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.269765 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:52:34.270920 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.253381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.1:49039)
0927 00:52:34.254071 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.254157 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.255646 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.256663 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.256677 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.257264 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:34.257860 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.257876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.259027 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.259034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.260856 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.260880 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.261551 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.261557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.261837 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.262757 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:34.262783 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:34.264960 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:34.267261 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.267268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.267286 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.267593 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.267946 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.267949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.267951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.268167 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.268813 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:34.268821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:34.269332 (+ 511us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.269544 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.269897 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":152,"threads_started":1}
I20260927 00:52:34.272490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1579
I20260927 00:52:34.272609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:34.272805 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37667
I20260927 00:52:34.272924 392 external_mini_cluster.cc:1428] Setting key 20f6b1711bd6904aa6ffdf70082878f0
I20260927 00:52:34.273093 1710 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:34.273713 1710 heartbeater.cc:511] Master 127.0.98.62:37667 requested a full tablet report, sending...
I20260927 00:52:34.275799 1507 ts_manager.cc:194] Registered new tserver with Master: dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889)
I20260927 00:52:34.277163 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49039
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:52:34.286284 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.278158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47797 (local address 127.0.98.62:37667)
0927 00:52:34.278290 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.278295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.278624 (+ 329us) server_negotiation.cc:407] Connection header received
0927 00:52:34.278747 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.278750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.278807 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.278871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.279447 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.280000 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.280839 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.281084 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.281171 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.282549 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.282553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.282556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.282580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.284140 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.284552 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.284556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.284558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.284639 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.285152 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.285156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.285158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.285473 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.285590 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.285899 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.286083 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:52:34.286296 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.278120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.1:47797)
0927 00:52:34.278532 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.278545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.278657 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.278910 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.278912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.279177 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:34.279339 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.279347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.280123 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.280126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.280731 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.280739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.281189 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.281193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.281296 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.281952 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:34.281964 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:34.282420 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:34.284239 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.284243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.284246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.284463 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.284826 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.284832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.284834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.285013 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.285616 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:34.285619 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:34.285723 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.285826 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.286132 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:34.304432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:34.421669 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:34.422048 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:34.422111 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:34.426836 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:34.426931 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:34.426977 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:52:34.427026 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:34.433512 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:34.435045 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:34.436427 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:52:34.448118 1719 server_base.cc:1034] running on GCE node
W20260927 00:52:34.448016 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:34.448028 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:34.448016 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:34.448864 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:34.449604 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:34.450774 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790470354450741 us; error 59 us; skew 500 ppm
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:34.453866 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:34.454843 1719 webserver.cc:460] Webserver started at http://127.0.98.2:33283/ using document root <none> and password file <none>
I20260927 00:52:34.455540 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:34.455613 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:34.455817 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:34.458000 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "c483b454c1e24f6a9a5cd5afe3daca63"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "2b0a8d7d9135d097cac081195fea05dd"
server_key_iv: "238322609a089c03de233636c1c6c034"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.458550 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c483b454c1e24f6a9a5cd5afe3daca63"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "2b0a8d7d9135d097cac081195fea05dd"
server_key_iv: "238322609a089c03de233636c1c6c034"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.462980 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:52:34.465883 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.467402 1719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:34.467523 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "c483b454c1e24f6a9a5cd5afe3daca63"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "2b0a8d7d9135d097cac081195fea05dd"
server_key_iv: "238322609a089c03de233636c1c6c034"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.467645 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:34.481109 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:34.484287 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:34.484463 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:34.485965 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:34.486020 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.486074 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:34.486104 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.497470 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42427
I20260927 00:52:34.497678 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42427 every 8 connection(s)
I20260927 00:52:34.498586 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:34.501981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1719
I20260927 00:52:34.502099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:34.502382 392 external_mini_cluster.cc:1428] Setting key 0120a757bb1ffabde0eaab3375c02ff7
I20260927 00:52:34.503297 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.487481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48397 (local address 127.0.98.62:37667)
0927 00:52:34.487644 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.487648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.489408 (+ 1760us) server_negotiation.cc:407] Connection header received
0927 00:52:34.490860 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.490865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.490941 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.491070 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.492921 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.493828 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.495630 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.495958 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.496103 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.499473 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.499477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.499480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.499510 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.500785 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.501490 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.501493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.501495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.501570 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.501918 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.501921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.501923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.502153 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.502259 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.502972 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.503112 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:52:34.504568 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.488024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.2:48397)
0927 00:52:34.489188 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.489259 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.490569 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.491324 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.491336 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.491907 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:34.492731 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.492753 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.494010 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.494014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.495439 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.495475 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.496322 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.496329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.496564 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.497348 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:34.497371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:34.499276 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:34.500926 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.500933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.500948 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.501376 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.501648 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.501651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.501652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.501796 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.502333 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:34.502340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:34.502671 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.502902 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.503340 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":143,"threads_started":1}
I20260927 00:52:34.506791 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37667
I20260927 00:52:34.506986 1850 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:34.507668 1850 heartbeater.cc:511] Master 127.0.98.62:37667 requested a full tablet report, sending...
I20260927 00:52:34.509155 1507 ts_manager.cc:194] Registered new tserver with Master: c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427)
I20260927 00:52:34.510010 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48397
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:52:34.518594 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.509763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46887 (local address 127.0.98.62:37667)
0927 00:52:34.509894 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.509898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.511533 (+ 1635us) server_negotiation.cc:407] Connection header received
0927 00:52:34.511659 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.511662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.511712 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.511787 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.512303 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.512847 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.513700 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.513929 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.514018 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.515261 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.515265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.515268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.515292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.516901 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.517335 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.517339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.517341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.517415 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.517659 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.517663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.517665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.517917 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.518006 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.518267 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.518402 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:52:34.518631 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.509671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.2:46887)
0927 00:52:34.511415 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.511431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.511564 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.511817 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.511819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.512059 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:34.512184 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.512190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.512995 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.512998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.513589 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.513597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.514038 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.514041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.514132 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.514686 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:34.514698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:34.515088 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:34.517030 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.517033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.517036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.517246 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.517496 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.517499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.517502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.517582 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.518007 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:34.518018 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:34.518115 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.518202 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.518443 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:34.532490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:34.644809 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:34.645071 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:34.645118 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:34.649334 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:34.649403 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:34.649443 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:52:34.649479 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:34.654894 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:34.656045 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:34.657092 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:34.667446 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:34.667448 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:34.667923 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:34.667975 1859 server_base.cc:1034] running on GCE node
I20260927 00:52:34.668414 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:34.669150 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:34.670341 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790470354670317 us; error 55 us; skew 500 ppm
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:34.673544 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:34.674640 1859 webserver.cc:460] Webserver started at http://127.0.98.3:36071/ using document root <none> and password file <none>
I20260927 00:52:34.675246 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:34.675318 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:34.675542 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:34.677623 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "dee0c90a351d455198357d1456c1fef8"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "39dd4f6eea921565344f921a5dba5e9a"
server_key_iv: "a9e664ae9df8617c37dddb8f0a96e61a"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.678158 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dee0c90a351d455198357d1456c1fef8"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "39dd4f6eea921565344f921a5dba5e9a"
server_key_iv: "a9e664ae9df8617c37dddb8f0a96e61a"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.682461 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:52:34.685423 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.686942 1859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:52:34.687192 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "dee0c90a351d455198357d1456c1fef8"
format_stamp: "Formatted at 2026-09-27 00:52:34 on dist-test-slave-9fh2"
server_key: "39dd4f6eea921565344f921a5dba5e9a"
server_key_iv: "a9e664ae9df8617c37dddb8f0a96e61a"
server_key_version: "encryptionkey@0"
I20260927 00:52:34.687319 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:34.727944 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:34.731024 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:34.731235 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:34.732769 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:34.732825 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.732882 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:34.732911 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:34.743841 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45733
I20260927 00:52:34.743956 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45733 every 8 connection(s)
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:34.744861 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:34.748970 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.734098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49791 (local address 127.0.98.62:37667)
0927 00:52:34.734245 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.734250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.735512 (+ 1262us) server_negotiation.cc:407] Connection header received
0927 00:52:34.737469 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.737473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.737550 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.737686 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.739552 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.740278 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.741692 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.741975 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.742079 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.745069 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.745072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.745075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.745105 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.746787 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.747351 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.747354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.747356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.747440 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.747767 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.747771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.747772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.747979 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.748065 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.748631 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.748770 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:52:34.749998 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.734553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.3:49791)
0927 00:52:34.735286 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.735364 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.737146 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.737942 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.737955 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.738629 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:34.739356 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.739374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.740425 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.740429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.741561 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.741572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.742135 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.742140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.742348 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.743048 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:34.743067 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:34.744862 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:34.746915 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.746921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.746940 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.747251 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.747523 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.747526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.747529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.747673 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.748074 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:34.748082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:34.748375 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.748558 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.748962 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":141,"threads_started":1}
I20260927 00:52:34.750142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 1859
I20260927 00:52:34.750269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:34.750509 392 external_mini_cluster.cc:1428] Setting key 13f76544c0b83f4f1e65b830779074b0
I20260927 00:52:34.751534 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37667
I20260927 00:52:34.751716 1990 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:34.752331 1990 heartbeater.cc:511] Master 127.0.98.62:37667 requested a full tablet report, sending...
I20260927 00:52:34.753521 1506 ts_manager.cc:194] Registered new tserver with Master: dee0c90a351d455198357d1456c1fef8 (127.0.98.3:45733)
I20260927 00:52:34.754191 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49791
I20260927 00:52:34.761900 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.754683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.3:34787)
0927 00:52:34.755125 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.755170 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.755305 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.755560 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.755563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.755808 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:34.755926 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.755933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.756744 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.756746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.757323 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.757331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.757755 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.757757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.757826 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.758422 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:34.758434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:34.758798 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:34.760586 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.760588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.760590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.760783 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.761011 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:34.761013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:34.761014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:34.761089 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:34.761462 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:34.761464 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:34.761544 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.761631 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.761741 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":154,"threads_started":1}
I20260927 00:52:34.761977 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.754848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34787 (local address 127.0.98.62:37667)
0927 00:52:34.754972 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.754976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.755263 (+ 287us) server_negotiation.cc:407] Connection header received
0927 00:52:34.755408 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.755410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.755449 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.755516 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.756076 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.756626 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.757436 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.757650 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.757728 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.758947 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.758950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.758952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.758975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.760474 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.760866 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.760869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.760870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.760941 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.761154 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.761156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.761158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.761380 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.761458 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.761721 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.761834 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:52:34.764606 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:52:34 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:34.793888 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.784502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.0.98.62:37667)
0927 00:52:34.784687 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.784692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.784710 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:52:34.784780 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.784783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.784834 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.784932 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:34.785682 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.786317 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.787318 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.787570 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.787659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.790302 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:34.790307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:34.790310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:34.790345 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:34.792042 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.792666 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.792669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.792671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.792757 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:34.792991 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:34.792993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:34.792994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:34.793196 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:34.793302 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.793639 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.793755 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:52:34.797669 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:34.800803 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:34.816747 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.810710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36889 (local address 127.0.0.1:58922)
0927 00:52:34.811116 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.811197 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.811352 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.812001 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.812008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.812032 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:34.812174 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.812183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.813486 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.813490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.814987 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.814999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.816066 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.816070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.816207 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.816262 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.816390 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.816545 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260927 00:52:34.817126 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.811597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42427 (local address 127.0.0.1:56830)
0927 00:52:34.812473 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.812486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.812629 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.813131 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.813134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.813153 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:34.813298 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.813305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.814397 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.814400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.815481 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.815490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.816482 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.816485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.816595 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.816626 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.816707 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.816797 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260927 00:52:34.817384 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.811836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.0.98.2:42427)
0927 00:52:34.812305 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.812316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.812562 (+ 246us) server_negotiation.cc:407] Connection header received
0927 00:52:34.812796 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.812802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.812984 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.813141 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:34.813400 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.814296 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.815614 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.816356 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.816456 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.816630 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.816769 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.817135 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":128,"threads_started":1}
I20260927 00:52:34.817420 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.810873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.0.98.1:36889)
0927 00:52:34.811238 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.811245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.811275 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:52:34.811510 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.811516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.811678 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.811816 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:34.812306 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.813357 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.815107 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.815927 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.816048 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.816216 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.817151 (+ 935us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.817283 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260927 00:52:34.817823 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.811935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.0.1:54352)
0927 00:52:34.812814 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.812826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.812893 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.813560 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.813564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.813581 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:34.813730 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.813737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.815105 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.815108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.816252 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.816263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.817432 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.817436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.817539 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.817564 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.817635 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.817707 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":30,"thread_start_us":101,"threads_started":1}
I20260927 00:52:34.817986 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.812155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.0.98.3:45733)
0927 00:52:34.813139 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.813149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.813169 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:52:34.813234 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.813239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.813421 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.813571 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:34.813853 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.814965 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.816375 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.817117 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.817214 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.817375 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.817696 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.817832 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":131,"threads_started":1}
I20260927 00:52:34.820449 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 66bcad9920474ade85d8ee0f07e9b209 (DEFAULT_TABLE table=test-table [id=374fe16de11642ce82b5479c72780323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:34.821406 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bcad9920474ade85d8ee0f07e9b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:34.821034 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 66bcad9920474ade85d8ee0f07e9b209 (DEFAULT_TABLE table=test-table [id=374fe16de11642ce82b5479c72780323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:34.821034 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 66bcad9920474ade85d8ee0f07e9b209 (DEFAULT_TABLE table=test-table [id=374fe16de11642ce82b5479c72780323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:34.822237 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bcad9920474ade85d8ee0f07e9b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:34.822232 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bcad9920474ade85d8ee0f07e9b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:34.827800 2013 tablet_bootstrap.cc:492] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Bootstrap starting.
I20260927 00:52:34.830318 2013 tablet_bootstrap.cc:654] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:34.831182 2015 tablet_bootstrap.cc:492] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Bootstrap starting.
I20260927 00:52:34.831128 2013 log.cc:824] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:34.831710 2014 tablet_bootstrap.cc:492] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Bootstrap starting.
I20260927 00:52:34.833660 2013 tablet_bootstrap.cc:492] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: No bootstrap required, opened a new log
I20260927 00:52:34.833843 2014 tablet_bootstrap.cc:654] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:34.833871 2013 ts_tablet_manager.cc:1397] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 00:52:34.834321 2015 tablet_bootstrap.cc:654] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:34.834699 2014 log.cc:824] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:34.835335 2015 log.cc:824] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:34.837338 2014 tablet_bootstrap.cc:492] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: No bootstrap required, opened a new log
I20260927 00:52:34.837404 2015 tablet_bootstrap.cc:492] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: No bootstrap required, opened a new log
I20260927 00:52:34.837579 2014 ts_tablet_manager.cc:1397] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:52:34.837579 2015 ts_tablet_manager.cc:1397] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260927 00:52:34.837952 2013 raft_consensus.cc:348] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.838188 2013 raft_consensus.cc:374] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:34.838236 2013 raft_consensus.cc:729] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb871022e0547d18e6cf5636a8eaeca, State: Initialized, Role: FOLLOWER
I20260927 00:52:34.838644 2013 consensus_queue.cc:260] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.839265 1710 heartbeater.cc:503] Master 127.0.98.62:37667 was elected leader, sending a full tablet report...
I20260927 00:52:34.839658 2013 ts_tablet_manager.cc:1428] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 00:52:34.842068 2014 raft_consensus.cc:348] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.842068 2015 raft_consensus.cc:348] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.842417 2015 raft_consensus.cc:374] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:34.842417 2014 raft_consensus.cc:374] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:34.842468 2015 raft_consensus.cc:729] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c483b454c1e24f6a9a5cd5afe3daca63, State: Initialized, Role: FOLLOWER
I20260927 00:52:34.842468 2014 raft_consensus.cc:729] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee0c90a351d455198357d1456c1fef8, State: Initialized, Role: FOLLOWER
I20260927 00:52:34.843020 2015 consensus_queue.cc:260] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.843020 2014 consensus_queue.cc:260] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.844138 1850 heartbeater.cc:503] Master 127.0.98.62:37667 was elected leader, sending a full tablet report...
I20260927 00:52:34.844307 2014 ts_tablet_manager.cc:1428] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260927 00:52:34.844307 2015 ts_tablet_manager.cc:1428] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 00:52:34.844658 1990 heartbeater.cc:503] Master 127.0.98.62:37667 was elected leader, sending a full tablet report...
I20260927 00:52:34.935752 2021 raft_consensus.cc:482] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:34.936058 2021 raft_consensus.cc:504] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.937530 2021 leader_election.cc:288] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889), dee0c90a351d455198357d1456c1fef8 (127.0.98.3:45733)
I20260927 00:52:34.942173 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.938008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58963 (local address 127.0.98.1:36889)
0927 00:52:34.938188 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.938192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.938213 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:52:34.938484 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.938488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.938553 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.938628 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:34.939004 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.939672 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.940867 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.941508 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.941610 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.941687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.941943 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.942038 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:52:34.942165 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.937864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36889 (local address 127.0.98.2:58963)
0927 00:52:34.938052 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.938073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.938219 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.938684 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.938688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.938717 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:34.938859 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.938867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.939787 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.939790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.940736 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.940747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.941631 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.941635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.941755 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.941786 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.941894 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.941995 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 00:52:34.942788 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.938139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.98.2:52253)
0927 00:52:34.938494 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:34.938506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:34.938612 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:34.938903 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:34.938906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:34.938927 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:34.939093 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.939100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.939880 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.939883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.941437 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:34.941449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.942313 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.942317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:34.942429 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.942457 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:52:34.942551 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:52:34.942625 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":153,"threads_started":1}
I20260927 00:52:34.942866 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.938371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52253 (local address 127.0.98.3:45733)
0927 00:52:34.938492 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.938495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.938595 (+ 100us) server_negotiation.cc:407] Connection header received
0927 00:52:34.938711 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.938714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.938761 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.938849 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:34.939262 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.939772 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.941577 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.942142 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.942244 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.942306 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.942571 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.942648 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:52:34.943235 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bcad9920474ade85d8ee0f07e9b209" candidate_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb871022e0547d18e6cf5636a8eaeca" is_pre_election: true
I20260927 00:52:34.943562 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bcad9920474ade85d8ee0f07e9b209" candidate_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee0c90a351d455198357d1456c1fef8" is_pre_election: true
I20260927 00:52:34.943663 1664 raft_consensus.cc:2454] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c483b454c1e24f6a9a5cd5afe3daca63 in term 0.
I20260927 00:52:34.943929 1944 raft_consensus.cc:2454] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c483b454c1e24f6a9a5cd5afe3daca63 in term 0.
I20260927 00:52:34.944311 1738 leader_election.cc:302] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c483b454c1e24f6a9a5cd5afe3daca63, dcb871022e0547d18e6cf5636a8eaeca; no voters:
I20260927 00:52:34.944595 2021 raft_consensus.cc:2781] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:34.944690 2021 raft_consensus.cc:482] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:34.944720 2021 raft_consensus.cc:3037] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:34.946031 2021 raft_consensus.cc:504] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.946501 2021 leader_election.cc:288] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [CANDIDATE]: Term 1 election: Requested vote from peers dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889), dee0c90a351d455198357d1456c1fef8 (127.0.98.3:45733)
I20260927 00:52:34.947046 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bcad9920474ade85d8ee0f07e9b209" candidate_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb871022e0547d18e6cf5636a8eaeca"
I20260927 00:52:34.947111 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bcad9920474ade85d8ee0f07e9b209" candidate_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee0c90a351d455198357d1456c1fef8"
I20260927 00:52:34.947233 1664 raft_consensus.cc:3037] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:34.947301 1944 raft_consensus.cc:3037] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:34.948504 1664 raft_consensus.cc:2454] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c483b454c1e24f6a9a5cd5afe3daca63 in term 1.
I20260927 00:52:34.948520 1944 raft_consensus.cc:2454] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c483b454c1e24f6a9a5cd5afe3daca63 in term 1.
I20260927 00:52:34.948798 1738 leader_election.cc:302] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c483b454c1e24f6a9a5cd5afe3daca63, dee0c90a351d455198357d1456c1fef8; no voters:
I20260927 00:52:34.948982 2021 raft_consensus.cc:2781] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:34.949247 2021 raft_consensus.cc:686] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 LEADER]: Becoming Leader. State: Replica: c483b454c1e24f6a9a5cd5afe3daca63, State: Running, Role: LEADER
I20260927 00:52:34.949622 2021 consensus_queue.cc:237] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:34.953300 1507 catalog_manager.cc:5495] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 reported cstate change: term changed from 0 to 1, leader changed from <none> to c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } health_report { overall_health: UNKNOWN } } }
I20260927 00:52:34.980903 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:34.976780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.0.98.2:42427)
0927 00:52:34.976909 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:34.976912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:34.976989 (+ 77us) server_negotiation.cc:407] Connection header received
0927 00:52:34.977232 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:34.977234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:34.977297 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:34.977392 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:34.977683 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.978217 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.979204 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:34.979453 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:34.979547 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:34.979916 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:34.980396 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:34.980493 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:34.980680 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:52:34.980737 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:52:34.988237 1664 raft_consensus.cc:1261] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Refusing update from remote peer c483b454c1e24f6a9a5cd5afe3daca63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:34.988495 1944 raft_consensus.cc:1261] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Refusing update from remote peer c483b454c1e24f6a9a5cd5afe3daca63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:34.989153 2021 consensus_queue.cc:1035] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:34.989385 2023 consensus_queue.cc:1035] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:52:34.996218 1991 tablet.cc:2369] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:52:35.003885 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333766574021820416 to 7333766573876740096. Current Snapshot: MvccSnapshot[applied={T|T < 7333766574021820416 or (T in {7333766574021820416})}]
I20260927 00:52:35.014083 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:35.014525 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:35.018581 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: Sending DeleteTablet for 3 replicas of tablet 66bcad9920474ade85d8ee0f07e9b209
I20260927 00:52:35.019732 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bcad9920474ade85d8ee0f07e9b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58922
I20260927 00:52:35.019919 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bcad9920474ade85d8ee0f07e9b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56830
I20260927 00:52:35.020315 2035 tablet_replica.cc:330] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: stopping tablet replica
I20260927 00:52:35.020723 2035 raft_consensus.cc:2229] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:35.020872 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bcad9920474ade85d8ee0f07e9b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54352
I20260927 00:52:35.020889 2036 tablet_replica.cc:330] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: stopping tablet replica
I20260927 00:52:35.021097 2035 raft_consensus.cc:2258] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:35.021483 2036 raft_consensus.cc:2229] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:35.021679 2037 tablet_replica.cc:330] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: stopping tablet replica
I20260927 00:52:35.022027 2037 raft_consensus.cc:2229] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:35.022058 2036 raft_consensus.cc:2258] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:35.022616 2037 raft_consensus.cc:2258] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:35.022914 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:35.023427 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:35.024658 2037 ts_tablet_manager.cc:1905] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:35.024900 2036 ts_tablet_manager.cc:1905] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:35.028141 2036 ts_tablet_manager.cc:1918] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:52:35.028332 2036 log.cc:1196] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/66bcad9920474ade85d8ee0f07e9b209
I20260927 00:52:35.028787 2036 ts_tablet_manager.cc:1939] T 66bcad9920474ade85d8ee0f07e9b209 P c483b454c1e24f6a9a5cd5afe3daca63: Deleting consensus metadata
I20260927 00:52:35.028802 2037 ts_tablet_manager.cc:1918] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:52:35.028909 2037 log.cc:1196] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/66bcad9920474ade85d8ee0f07e9b209
I20260927 00:52:35.029225 2035 ts_tablet_manager.cc:1905] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:35.029261 2037 ts_tablet_manager.cc:1939] T 66bcad9920474ade85d8ee0f07e9b209 P dee0c90a351d455198357d1456c1fef8: Deleting consensus metadata
I20260927 00:52:35.029276 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 0c105d4a94a54e0da4fefa542f48ac18 (DEFAULT_TABLE table=test-table [id=534fa1f5c63d40d6bbc480983ef15592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:35.029560 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c105d4a94a54e0da4fefa542f48ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:35.031392 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 0c105d4a94a54e0da4fefa542f48ac18 (DEFAULT_TABLE table=test-table [id=534fa1f5c63d40d6bbc480983ef15592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:35.031667 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c105d4a94a54e0da4fefa542f48ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:35.032377 2035 ts_tablet_manager.cc:1918] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:52:35.032497 2035 log.cc:1196] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/66bcad9920474ade85d8ee0f07e9b209
I20260927 00:52:35.032796 2035 ts_tablet_manager.cc:1939] T 66bcad9920474ade85d8ee0f07e9b209 P dcb871022e0547d18e6cf5636a8eaeca: Deleting consensus metadata
I20260927 00:52:35.033680 2015 tablet_bootstrap.cc:492] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Bootstrap starting.
I20260927 00:52:35.033927 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 0c105d4a94a54e0da4fefa542f48ac18 (DEFAULT_TABLE table=test-table [id=534fa1f5c63d40d6bbc480983ef15592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:35.034157 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c105d4a94a54e0da4fefa542f48ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:35.034672 2015 tablet_bootstrap.cc:654] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:35.036299 2015 tablet_bootstrap.cc:492] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: No bootstrap required, opened a new log
I20260927 00:52:35.036414 2015 ts_tablet_manager.cc:1397] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:52:35.036772 1494 catalog_manager.cc:4841] TS dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889): tablet 66bcad9920474ade85d8ee0f07e9b209 (table test-table [id=374fe16de11642ce82b5479c72780323]) successfully deleted
I20260927 00:52:35.036993 1494 catalog_manager.cc:4841] TS c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427): tablet 66bcad9920474ade85d8ee0f07e9b209 (table test-table [id=374fe16de11642ce82b5479c72780323]) successfully deleted
I20260927 00:52:35.037708 1494 catalog_manager.cc:4841] TS dee0c90a351d455198357d1456c1fef8 (127.0.98.3:45733): tablet 66bcad9920474ade85d8ee0f07e9b209 (table test-table [id=374fe16de11642ce82b5479c72780323]) successfully deleted
I20260927 00:52:35.037741 2015 raft_consensus.cc:348] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.037845 2015 raft_consensus.cc:374] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:35.037876 2015 raft_consensus.cc:729] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c483b454c1e24f6a9a5cd5afe3daca63, State: Initialized, Role: FOLLOWER
I20260927 00:52:35.037997 2015 consensus_queue.cc:260] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.038257 2015 ts_tablet_manager.cc:1428] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:52:35.039247 2014 tablet_bootstrap.cc:492] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Bootstrap starting.
I20260927 00:52:35.040203 2014 tablet_bootstrap.cc:654] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:35.042583 2013 tablet_bootstrap.cc:492] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Bootstrap starting.
I20260927 00:52:35.043627 2013 tablet_bootstrap.cc:654] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:35.044581 2014 tablet_bootstrap.cc:492] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: No bootstrap required, opened a new log
I20260927 00:52:35.044672 2014 ts_tablet_manager.cc:1397] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 00:52:35.045112 2014 raft_consensus.cc:348] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.045253 2014 raft_consensus.cc:374] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:35.045287 2014 raft_consensus.cc:729] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee0c90a351d455198357d1456c1fef8, State: Initialized, Role: FOLLOWER
I20260927 00:52:35.045439 2014 consensus_queue.cc:260] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.046197 2013 tablet_bootstrap.cc:492] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: No bootstrap required, opened a new log
I20260927 00:52:35.046286 2013 ts_tablet_manager.cc:1397] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 00:52:35.047019 2013 raft_consensus.cc:348] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.047436 2014 ts_tablet_manager.cc:1428] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:52:35.047575 2013 raft_consensus.cc:374] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:35.047626 2013 raft_consensus.cc:729] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb871022e0547d18e6cf5636a8eaeca, State: Initialized, Role: FOLLOWER
I20260927 00:52:35.047765 2013 consensus_queue.cc:260] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.048012 2013 ts_tablet_manager.cc:1428] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:52:35.195657 2020 raft_consensus.cc:482] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:35.195905 2020 raft_consensus.cc:504] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.197330 2020 leader_election.cc:288] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889), c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427)
I20260927 00:52:35.202615 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.198150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42503 (local address 127.0.98.2:42427)
0927 00:52:35.198344 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.198349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.198439 (+ 90us) server_negotiation.cc:407] Connection header received
0927 00:52:35.198577 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.198579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.198620 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.198694 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:35.198926 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.199647 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.201173 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.201876 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.201977 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.202095 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.202374 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.202451 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:52:35.202615 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.197826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33125 (local address 127.0.98.1:36889)
0927 00:52:35.197968 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.197972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.197989 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:52:35.198168 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.198172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.198244 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.198342 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:35.198691 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.199543 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.201173 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.201859 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.201977 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.202071 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.202369 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.202451 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:52:35.202620 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.197669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36889 (local address 127.0.98.3:33125)
0927 00:52:35.197848 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:35.197866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:35.198020 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:35.198381 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:35.198384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:35.198404 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:35.198580 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.198587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.199671 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.199673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.201026 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.201038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.202022 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.202026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.202212 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.202253 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:52:35.202352 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:52:35.202420 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":58,"spinlock_wait_cycles":768}
I20260927 00:52:35.202633 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.197996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42427 (local address 127.0.98.3:42503)
0927 00:52:35.198322 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:35.198337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:35.198470 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:35.198717 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:35.198720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:35.198736 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:35.198825 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.198829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.199750 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.199753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.201028 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.201038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.202029 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.202032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.202238 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.202262 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:52:35.202339 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:52:35.202420 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":119,"spinlock_wait_cycles":3200,"thread_start_us":90,"threads_started":1}
I20260927 00:52:35.203341 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c105d4a94a54e0da4fefa542f48ac18" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb871022e0547d18e6cf5636a8eaeca" is_pre_election: true
I20260927 00:52:35.203504 1664 raft_consensus.cc:2454] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 0.
I20260927 00:52:35.203482 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c105d4a94a54e0da4fefa542f48ac18" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" is_pre_election: true
I20260927 00:52:35.203791 1804 raft_consensus.cc:2454] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 0.
I20260927 00:52:35.203898 1878 leader_election.cc:302] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcb871022e0547d18e6cf5636a8eaeca, dee0c90a351d455198357d1456c1fef8; no voters:
I20260927 00:52:35.204257 2020 raft_consensus.cc:2781] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:35.204329 2020 raft_consensus.cc:482] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:35.204372 2020 raft_consensus.cc:3037] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:35.205626 2020 raft_consensus.cc:504] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.206104 2020 leader_election.cc:288] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 election: Requested vote from peers dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889), c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427)
I20260927 00:52:35.206411 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c105d4a94a54e0da4fefa542f48ac18" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb871022e0547d18e6cf5636a8eaeca"
I20260927 00:52:35.206547 1664 raft_consensus.cc:3037] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:35.206533 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c105d4a94a54e0da4fefa542f48ac18" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c483b454c1e24f6a9a5cd5afe3daca63"
I20260927 00:52:35.206688 1804 raft_consensus.cc:3037] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:35.207680 1664 raft_consensus.cc:2454] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 1.
I20260927 00:52:35.208043 1878 leader_election.cc:302] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcb871022e0547d18e6cf5636a8eaeca, dee0c90a351d455198357d1456c1fef8; no voters:
I20260927 00:52:35.208135 1804 raft_consensus.cc:2454] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 1.
I20260927 00:52:35.208246 2020 raft_consensus.cc:2781] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:35.208447 2020 raft_consensus.cc:686] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 1 LEADER]: Becoming Leader. State: Replica: dee0c90a351d455198357d1456c1fef8, State: Running, Role: LEADER
I20260927 00:52:35.208789 2020 consensus_queue.cc:237] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.212075 1507 catalog_manager.cc:5495] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee0c90a351d455198357d1456c1fef8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } health_report { overall_health: HEALTHY } } }
W20260927 00:52:35.250679 1851 tablet.cc:2369] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:52:35.268647 1711 tablet.cc:2369] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:52:35 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:35.286301 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.269283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.0.98.62:37667)
0927 00:52:35.269494 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.269499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.269676 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:52:35.269837 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.269840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.269880 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.269947 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:35.270728 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.271348 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.272181 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.272390 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.272479 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.282815 (+ 10336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:35.282819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:35.282821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:35.282846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:35.284456 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:35.284998 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:35.285002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:35.285004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:35.285079 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:35.285314 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:35.285318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:35.285319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:35.285544 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:35.285632 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.285963 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.286098 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 00:52:35.286316 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.269227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.0.1:47846)
0927 00:52:35.269552 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:35.269566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:35.269735 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:35.269982 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:35.269985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:35.270498 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:35.270621 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.270628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.271449 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.271453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.272075 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.272083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.272518 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.272522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.272596 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.280553 (+ 7957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:35.280573 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:35.282580 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:35.284597 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:35.284613 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:35.284622 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:35.284902 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:35.285158 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:35.285161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:35.285163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:35.285239 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:35.285646 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:35.285656 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:35.285799 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 00:52:35.285913 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:52:35.286183 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":83,"thread_start_us":106,"threads_started":1}
I20260927 00:52:35.291985 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:52:35.292928 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:35.298514 1644 tablet_service.cc:1467] Processing CreateTablet for tablet b96a2acc9f384d73abc319c62d5c44fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1616f8d832e24853a502a70d6c6c8bea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:52:35.298514 1784 tablet_service.cc:1467] Processing CreateTablet for tablet b96a2acc9f384d73abc319c62d5c44fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1616f8d832e24853a502a70d6c6c8bea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:52:35.298605 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b96a2acc9f384d73abc319c62d5c44fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1616f8d832e24853a502a70d6c6c8bea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:52:35.298923 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96a2acc9f384d73abc319c62d5c44fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:35.298923 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96a2acc9f384d73abc319c62d5c44fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:35.299093 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96a2acc9f384d73abc319c62d5c44fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:35.302074 2013 tablet_bootstrap.cc:492] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: Bootstrap starting.
I20260927 00:52:35.302091 2015 tablet_bootstrap.cc:492] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: Bootstrap starting.
I20260927 00:52:35.302924 2013 tablet_bootstrap.cc:654] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:35.302974 2015 tablet_bootstrap.cc:654] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:35.303611 2014 tablet_bootstrap.cc:492] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: Bootstrap starting.
I20260927 00:52:35.304121 2015 tablet_bootstrap.cc:492] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: No bootstrap required, opened a new log
I20260927 00:52:35.304203 2015 ts_tablet_manager.cc:1397] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:52:35.304482 2014 tablet_bootstrap.cc:654] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:35.304601 2015 raft_consensus.cc:348] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.304721 2015 raft_consensus.cc:374] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:35.304742 2015 raft_consensus.cc:729] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c483b454c1e24f6a9a5cd5afe3daca63, State: Initialized, Role: FOLLOWER
I20260927 00:52:35.304877 2015 consensus_queue.cc:260] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.305130 2015 ts_tablet_manager.cc:1428] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:52:35.305370 2013 tablet_bootstrap.cc:492] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: No bootstrap required, opened a new log
I20260927 00:52:35.305325 2021 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.305454 2013 ts_tablet_manager.cc:1397] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:35.305452 2021 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.305876 2013 raft_consensus.cc:348] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.305991 2013 raft_consensus.cc:374] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:35.306021 2013 raft_consensus.cc:729] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb871022e0547d18e6cf5636a8eaeca, State: Initialized, Role: FOLLOWER
I20260927 00:52:35.306126 2014 tablet_bootstrap.cc:492] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: No bootstrap required, opened a new log
I20260927 00:52:35.306156 2013 consensus_queue.cc:260] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.306231 2014 ts_tablet_manager.cc:1397] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:52:35.306438 2013 ts_tablet_manager.cc:1428] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:52:35.306692 2014 raft_consensus.cc:348] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.306828 2014 raft_consensus.cc:374] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:35.306860 2014 raft_consensus.cc:729] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee0c90a351d455198357d1456c1fef8, State: Initialized, Role: FOLLOWER
I20260927 00:52:35.306774 2019 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.306950 2019 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.306995 2014 consensus_queue.cc:260] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.307261 2014 ts_tablet_manager.cc:1428] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:52:35.307999 2020 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.308136 2020 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.489127 2020 raft_consensus.cc:482] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:35.489305 2020 raft_consensus.cc:504] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.489743 2020 leader_election.cc:288] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889), c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427)
I20260927 00:52:35.490473 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a2acc9f384d73abc319c62d5c44fa" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" is_pre_election: true
I20260927 00:52:35.490482 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a2acc9f384d73abc319c62d5c44fa" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb871022e0547d18e6cf5636a8eaeca" is_pre_election: true
I20260927 00:52:35.490703 1804 raft_consensus.cc:2454] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 0.
I20260927 00:52:35.490703 1664 raft_consensus.cc:2454] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 0.
I20260927 00:52:35.491081 1878 leader_election.cc:302] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcb871022e0547d18e6cf5636a8eaeca, dee0c90a351d455198357d1456c1fef8; no voters:
I20260927 00:52:35.491323 2020 raft_consensus.cc:2781] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:35.491386 2020 raft_consensus.cc:482] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:35.491438 2020 raft_consensus.cc:3037] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:35.492463 2020 raft_consensus.cc:504] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.492831 2020 leader_election.cc:288] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 election: Requested vote from peers dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889), c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427)
I20260927 00:52:35.493276 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a2acc9f384d73abc319c62d5c44fa" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb871022e0547d18e6cf5636a8eaeca"
I20260927 00:52:35.493361 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a2acc9f384d73abc319c62d5c44fa" candidate_uuid: "dee0c90a351d455198357d1456c1fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c483b454c1e24f6a9a5cd5afe3daca63"
I20260927 00:52:35.493445 1664 raft_consensus.cc:3037] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:35.493520 1804 raft_consensus.cc:3037] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:35.494516 1804 raft_consensus.cc:2454] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 1.
I20260927 00:52:35.494674 1664 raft_consensus.cc:2454] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee0c90a351d455198357d1456c1fef8 in term 1.
I20260927 00:52:35.494824 1878 leader_election.cc:302] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c483b454c1e24f6a9a5cd5afe3daca63, dee0c90a351d455198357d1456c1fef8; no voters:
I20260927 00:52:35.495014 2020 raft_consensus.cc:2781] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:35.495124 2020 raft_consensus.cc:686] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [term 1 LEADER]: Becoming Leader. State: Replica: dee0c90a351d455198357d1456c1fef8, State: Running, Role: LEADER
I20260927 00:52:35.495308 2020 consensus_queue.cc:237] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } }
I20260927 00:52:35.495968 2042 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: TxnStatusTablet state changed. Reason: New leader dee0c90a351d455198357d1456c1fef8. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.496076 2042 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:52:35.496296 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:52:35.496438 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:52:35.497247 1507 catalog_manager.cc:5495] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee0c90a351d455198357d1456c1fef8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } health_report { overall_health: HEALTHY } } }
I20260927 00:52:35.553879 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.546358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.0.1:54356)
0927 00:52:35.546666 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:35.546686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:35.546826 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:35.547321 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:35.547324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:35.547338 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:35.547484 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.547490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.548511 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.548515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.549653 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.549665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.550541 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.550547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.550714 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.550785 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:35.553463 (+ 2678us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:35.553500 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:52:35.553578 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:52:35.553711 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260927 00:52:35.553894 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.546468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.0.98.3:45733)
0927 00:52:35.546748 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.546753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.546774 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:52:35.547045 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.547049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.547130 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.547307 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:35.547603 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.548388 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.549803 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.550186 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.550295 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.550979 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:35.552396 (+ 1417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:35.552509 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.553622 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.553713 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260927 00:52:35.556963 1804 raft_consensus.cc:1261] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Refusing update from remote peer dee0c90a351d455198357d1456c1fef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:35.557013 1664 raft_consensus.cc:1261] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Refusing update from remote peer dee0c90a351d455198357d1456c1fef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:35.557497 2042 consensus_queue.cc:1035] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:35.557796 2020 consensus_queue.cc:1035] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:35.559326 2021 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: TxnStatusTablet state changed. Reason: New leader dee0c90a351d455198357d1456c1fef8. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.559433 2021 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.559417 2019 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: TxnStatusTablet state changed. Reason: New leader dee0c90a351d455198357d1456c1fef8. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.559578 2019 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.561347 2020 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.561466 2020 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:52:35.562987 2059 mvcc.cc:205] Tried to move back new op lower bound from 7333766576355594240 to 7333766576110096384. Current Snapshot: MvccSnapshot[applied={T|T < 7333766576355594240}]
I20260927 00:52:35.563210 2021 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.563303 2021 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P c483b454c1e24f6a9a5cd5afe3daca63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.568567 2042 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.568686 2042 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dee0c90a351d455198357d1456c1fef8: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:52:35.569664 2019 tablet_replica.cc:439] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dee0c90a351d455198357d1456c1fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 } } peers { permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "dee0c90a351d455198357d1456c1fef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } }
I20260927 00:52:35.569847 2019 tablet_replica.cc:442] T b96a2acc9f384d73abc319c62d5c44fa P dcb871022e0547d18e6cf5636a8eaeca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:35.572520 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.566766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.98.3:45733)
0927 00:52:35.566889 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.566893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.567066 (+ 173us) server_negotiation.cc:407] Connection header received
0927 00:52:35.567233 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.567236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.567282 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.567343 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:35.567576 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.568080 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.571433 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.571656 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.571719 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.572098 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:35.572177 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:35.572240 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.572344 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.572397 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:52:35.584666 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.580343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33871 (local address 127.0.98.3:45733)
0927 00:52:35.580450 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.580453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.580813 (+ 360us) server_negotiation.cc:407] Connection header received
0927 00:52:35.580958 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.580961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.581004 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.581073 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:35.581417 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.581980 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.583338 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.583624 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.583723 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.584106 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:35.584188 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:35.584266 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.584395 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.584487 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":35}
I20260927 00:52:35.584757 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.580251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.98.3:33871)
0927 00:52:35.580655 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:35.580672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:35.580833 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:35.581103 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:35.581106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:35.581114 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:35.581268 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.581274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.582079 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.582082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.583192 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:35.583203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.583753 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.583756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:35.583859 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.583926 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:35.584276 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:35.584283 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:52:35.584350 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:52:35.584478 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260927 00:52:35.591831 1664 raft_consensus.cc:1261] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Refusing update from remote peer dee0c90a351d455198357d1456c1fef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:35.591836 1804 raft_consensus.cc:1261] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Refusing update from remote peer dee0c90a351d455198357d1456c1fef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:35.592253 2020 consensus_queue.cc:1035] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c483b454c1e24f6a9a5cd5afe3daca63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:35.592440 2067 consensus_queue.cc:1035] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb871022e0547d18e6cf5636a8eaeca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:35.596421 2058 mvcc.cc:205] Tried to move back new op lower bound from 7333766576499044352 to 7333766574938292224. Current Snapshot: MvccSnapshot[applied={T|T < 7333766576499044352}]
I20260927 00:52:35.641956 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:35.642207 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:35.644783 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0482dc7f9e44f3196ddf00731a440e0: Sending DeleteTablet for 3 replicas of tablet 0c105d4a94a54e0da4fefa542f48ac18
I20260927 00:52:35.645457 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c105d4a94a54e0da4fefa542f48ac18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58922
I20260927 00:52:35.645677 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c105d4a94a54e0da4fefa542f48ac18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56830
I20260927 00:52:35.645846 2075 tablet_replica.cc:330] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: stopping tablet replica
I20260927 00:52:35.645968 2075 raft_consensus.cc:2229] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:35.646090 2075 raft_consensus.cc:2258] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:35.646461 2076 tablet_replica.cc:330] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: stopping tablet replica
I20260927 00:52:35.646492 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c105d4a94a54e0da4fefa542f48ac18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54352
I20260927 00:52:35.646692 2076 raft_consensus.cc:2229] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:35.646751 2075 ts_tablet_manager.cc:1905] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:35.646893 2076 raft_consensus.cc:2258] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:35.647620 2076 ts_tablet_manager.cc:1905] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:35.648254 2077 tablet_replica.cc:330] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: stopping tablet replica
I20260927 00:52:35.648418 2077 raft_consensus.cc:2229] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:35.648855 2077 raft_consensus.cc:2258] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:35.649744 2075 ts_tablet_manager.cc:1918] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:35.649828 2075 log.cc:1196] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/0c105d4a94a54e0da4fefa542f48ac18
I20260927 00:52:35.650072 2075 ts_tablet_manager.cc:1939] T 0c105d4a94a54e0da4fefa542f48ac18 P dcb871022e0547d18e6cf5636a8eaeca: Deleting consensus metadata
I20260927 00:52:35.650795 2077 ts_tablet_manager.cc:1905] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:35.650790 1494 catalog_manager.cc:4841] TS dcb871022e0547d18e6cf5636a8eaeca (127.0.98.1:36889): tablet 0c105d4a94a54e0da4fefa542f48ac18 (table test-table [id=534fa1f5c63d40d6bbc480983ef15592]) successfully deleted
I20260927 00:52:35.651001 2076 ts_tablet_manager.cc:1918] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:35.651085 2076 log.cc:1196] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/0c105d4a94a54e0da4fefa542f48ac18
I20260927 00:52:35.651319 2076 ts_tablet_manager.cc:1939] T 0c105d4a94a54e0da4fefa542f48ac18 P c483b454c1e24f6a9a5cd5afe3daca63: Deleting consensus metadata
I20260927 00:52:35.652041 1494 catalog_manager.cc:4841] TS c483b454c1e24f6a9a5cd5afe3daca63 (127.0.98.2:42427): tablet 0c105d4a94a54e0da4fefa542f48ac18 (table test-table [id=534fa1f5c63d40d6bbc480983ef15592]) successfully deleted
I20260927 00:52:35.654333 2077 ts_tablet_manager.cc:1918] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:35.654424 2077 log.cc:1196] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/0c105d4a94a54e0da4fefa542f48ac18
I20260927 00:52:35.654676 2077 ts_tablet_manager.cc:1939] T 0c105d4a94a54e0da4fefa542f48ac18 P dee0c90a351d455198357d1456c1fef8: Deleting consensus metadata
I20260927 00:52:35.655555 1494 catalog_manager.cc:4841] TS dee0c90a351d455198357d1456c1fef8 (127.0.98.3:45733): tablet 0c105d4a94a54e0da4fefa542f48ac18 (table test-table [id=534fa1f5c63d40d6bbc480983ef15592]) successfully deleted
Sep 27 00:52:35 dist-test-slave-9fh2 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470354, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:35.665536 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.649450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.0.98.1:36889)
0927 00:52:35.649595 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.649600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.655475 (+ 5875us) server_negotiation.cc:407] Connection header received
0927 00:52:35.655668 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.655671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.655718 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.655786 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:35.656463 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.656985 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.657768 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.657979 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.658063 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.661731 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:35.661742 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:35.661759 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:35.661787 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:35.663454 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:35.664116 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:35.664121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:35.664122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:35.664199 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:35.664465 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:35.664467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:35.664468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:35.664720 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:35.664879 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.665222 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.665351 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:52:35.666183 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58932
I20260927 00:52:35.675274 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:35.668433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.62:37667)
0927 00:52:35.668560 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:35.668565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:35.668733 (+ 168us) server_negotiation.cc:407] Connection header received
0927 00:52:35.668855 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:35.668858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:35.668904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:35.668967 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:35.669587 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.670106 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.670857 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:35.671127 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:35.671245 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:35.672503 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:35.672507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:35.672510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:35.672534 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:35.673796 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:35.674156 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:35.674160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:35.674162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:35.674239 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:35.674478 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:35.674481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:35.674483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:35.674652 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:35.674771 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:35.674972 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:52:35.675076 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:52:35.675760 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47852
I20260927 00:52:35.677316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1579
I20260927 00:52:35.684532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1719
I20260927 00:52:35.694089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1859
I20260927 00:52:35.700994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 1475
2026-09-27T00:52:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3032 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:35 dist-test-slave-9fh2 krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:35 dist-test-slave-9fh2 krb5kdc[2084](info): set up 2 sockets
Sep 27 00:52:35 dist-test-slave-9fh2 krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260927 00:52:36.671226 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:36 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:36.802134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:36.930624 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:36.930994 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:36.931056 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:36.935420 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:36.935516 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:52:36.935540 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:36.935566 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:36.935586 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:36.935608 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:36.941695 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43023
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:36.943223 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:36.944509 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:36.954846 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:36.954852 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:36.954838 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:36.955031 2097 server_base.cc:1034] running on GCE node
I20260927 00:52:36.956086 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:36.957510 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:36.958702 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790470356958662 us; error 62 us; skew 500 ppm
Sep 27 00:52:36 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:36.962057 2097 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:36.963352 2097 webserver.cc:460] Webserver started at http://127.0.98.62:37401/ using document root <none> and password file <none>
I20260927 00:52:36.964033 2097 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:36.964107 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:36.964355 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:36.966876 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c1238144f3e3426e80b128f198d1e04a"
format_stamp: "Formatted at 2026-09-27 00:52:36 on dist-test-slave-9fh2"
server_key: "e7e87247cad7cf529b25bd0cabea9aac"
server_key_iv: "decd2a525c9a8bdfee63efdeadb78e92"
server_key_version: "encryptionkey@0"
I20260927 00:52:36.967545 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1238144f3e3426e80b128f198d1e04a"
format_stamp: "Formatted at 2026-09-27 00:52:36 on dist-test-slave-9fh2"
server_key: "e7e87247cad7cf529b25bd0cabea9aac"
server_key_iv: "decd2a525c9a8bdfee63efdeadb78e92"
server_key_version: "encryptionkey@0"
I20260927 00:52:36.972163 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:36.975446 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:36.977098 2097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:36.977309 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c1238144f3e3426e80b128f198d1e04a"
format_stamp: "Formatted at 2026-09-27 00:52:36 on dist-test-slave-9fh2"
server_key: "e7e87247cad7cf529b25bd0cabea9aac"
server_key_iv: "decd2a525c9a8bdfee63efdeadb78e92"
server_key_version: "encryptionkey@0"
I20260927 00:52:36.977456 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:36.989040 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:36.993042 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:36.993270 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:37.003037 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43023
I20260927 00:52:37.003062 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43023 every 8 connection(s)
I20260927 00:52:37.004295 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:37.007726 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:37.009963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 2097
I20260927 00:52:37.010088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:37.010365 392 external_mini_cluster.cc:1428] Setting key cdc2586de0fde578b10f972681c0b086
I20260927 00:52:37.014742 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Bootstrap starting.
I20260927 00:52:37.017482 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:37.018252 2175 log.cc:824] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:37.020416 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: No bootstrap required, opened a new log
Sep 27 00:52:37 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:37.024426 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1238144f3e3426e80b128f198d1e04a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43023 } }
I20260927 00:52:37.024748 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:37.024791 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1238144f3e3426e80b128f198d1e04a, State: Initialized, Role: FOLLOWER
I20260927 00:52:37.025306 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1238144f3e3426e80b128f198d1e04a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43023 } }
I20260927 00:52:37.025483 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:37.025556 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:37.025661 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:37.026870 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1238144f3e3426e80b128f198d1e04a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43023 } }
I20260927 00:52:37.027310 2175 leader_election.cc:302] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1238144f3e3426e80b128f198d1e04a; no voters:
I20260927 00:52:37.027678 2175 leader_election.cc:288] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:37.027871 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:37.027938 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.012269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.0.98.62:43023)
0927 00:52:37.012908 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:37.012930 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:37.012983 (+ 53us) server_negotiation.cc:407] Connection header received
0927 00:52:37.013950 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:37.013975 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:37.014509 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:37.014951 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:37.015932 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.017124 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.018084 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.018669 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.018843 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.021927 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:37.021933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:37.021953 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:37.021989 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:37.023963 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.024637 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.024643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.024649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.024819 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.025198 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.025204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.025206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.025873 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:37.026107 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:37.026416 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:52:37.026797 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":181,"threads_started":1}
I20260927 00:52:37.028136 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [term 1 LEADER]: Becoming Leader. State: Replica: c1238144f3e3426e80b128f198d1e04a, State: Running, Role: LEADER
I20260927 00:52:37.028609 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1238144f3e3426e80b128f198d1e04a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43023 } }
I20260927 00:52:37.029090 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:37.031180 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1238144f3e3426e80b128f198d1e04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1238144f3e3426e80b128f198d1e04a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43023 } } }
I20260927 00:52:37.031737 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:37.031435 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1238144f3e3426e80b128f198d1e04a. Latest consensus state: current_term: 1 leader_uuid: "c1238144f3e3426e80b128f198d1e04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1238144f3e3426e80b128f198d1e04a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43023 } } }
I20260927 00:52:37.031987 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:37.034462 2194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:37.034571 2194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:37.034672 2195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:37.035630 2195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:37.042872 2195 catalog_manager.cc:1347] Generated new cluster ID: 858fd3c55d4f42e1a4d26734bf24aa2e
I20260927 00:52:37.042977 2195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:37.064795 2195 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:37.065990 2195 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:37.074057 2195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Generated new TSK 0
I20260927 00:52:37.074896 2195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:37.114655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43023
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:37.248294 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:37.248687 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:37.248744 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:37.253324 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:37.253423 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:37.253473 2201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:52:37.253489 2201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:52:37.253523 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:37.259869 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:37.261325 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:37.262627 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:37.275061 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:37.275045 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:37.275045 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:37.275583 2201 server_base.cc:1034] running on GCE node
I20260927 00:52:37.276145 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:37.277030 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:37.278214 2201 hybrid_clock.cc:648] HybridClock initialized: now 1790470357278196 us; error 41 us; skew 500 ppm
Sep 27 00:52:37 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:37.281670 2201 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:37.282927 2201 webserver.cc:460] Webserver started at http://127.0.98.1:35783/ using document root <none> and password file <none>
I20260927 00:52:37.283651 2201 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:37.283721 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:37.283957 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:37.286465 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7"
format_stamp: "Formatted at 2026-09-27 00:52:37 on dist-test-slave-9fh2"
server_key: "9f0bbe55dcf1de91f793218be44f75bd"
server_key_iv: "456196c20f6300ba99efa671a9edd67a"
server_key_version: "encryptionkey@0"
I20260927 00:52:37.287110 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7"
format_stamp: "Formatted at 2026-09-27 00:52:37 on dist-test-slave-9fh2"
server_key: "9f0bbe55dcf1de91f793218be44f75bd"
server_key_iv: "456196c20f6300ba99efa671a9edd67a"
server_key_version: "encryptionkey@0"
I20260927 00:52:37.291909 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:37.295094 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:37.297135 2201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:37.297353 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7"
format_stamp: "Formatted at 2026-09-27 00:52:37 on dist-test-slave-9fh2"
server_key: "9f0bbe55dcf1de91f793218be44f75bd"
server_key_iv: "456196c20f6300ba99efa671a9edd67a"
server_key_version: "encryptionkey@0"
I20260927 00:52:37.297518 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:37.310571 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:37.314960 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:37.315248 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:37.317535 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:37.317615 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:37.317673 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:37.317691 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:37.330999 2201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39385
Sep 27 00:52:37 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:37.331211 2331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39385 every 8 connection(s)
I20260927 00:52:37.332266 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:37.333079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 2201
I20260927 00:52:37.333273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:37.333586 392 external_mini_cluster.cc:1428] Setting key b521947ff6dbf4bbddb90ba1ce655f97
I20260927 00:52:37.336483 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.319217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58041 (local address 127.0.98.62:43023)
0927 00:52:37.319368 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:37.319373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:37.319837 (+ 464us) server_negotiation.cc:407] Connection header received
0927 00:52:37.322121 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:37.322125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:37.322206 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:37.322334 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:37.324686 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.325438 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.327674 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.327957 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.328074 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.332107 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:37.332112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:37.332114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:37.332144 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:37.333954 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.334649 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.334653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.334655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.334735 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.335074 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.335078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.335080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.335383 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:37.335486 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:37.336164 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 00:52:37.336323 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:52:37.337620 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.318830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.98.1:58041)
0927 00:52:37.319576 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:37.319655 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:37.321021 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:37.322597 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:37.322610 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:37.323272 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:37.324485 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.324516 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.325584 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.325588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.327458 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.327473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.328220 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.328226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.328650 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.329596 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:37.329621 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:37.331863 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:37.334113 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.334120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.334138 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.334524 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.334821 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.334824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.334826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.334972 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.335514 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:37.335521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:37.335847 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 00:52:37.336069 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 00:52:37.336449 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":151,"threads_started":1}
I20260927 00:52:37.339191 2332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43023
I20260927 00:52:37.339370 2332 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:37.339931 2332 heartbeater.cc:511] Master 127.0.98.62:43023 requested a full tablet report, sending...
I20260927 00:52:37.342026 2128 ts_manager.cc:194] Registered new tserver with Master: ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385)
I20260927 00:52:37.344231 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58041
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:37.352370 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.342370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.98.1:54777)
0927 00:52:37.342744 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:37.342763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:37.342899 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:37.343602 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:37.343607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:37.343853 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:37.344028 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.344036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.345095 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.345099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.346029 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.346043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.346700 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.346704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.346802 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.347485 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:37.347498 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:37.347936 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:37.349815 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.349829 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.349831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.350119 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.351122 (+ 1003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.351125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.351126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.351256 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.351796 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:37.351799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:37.351909 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 00:52:37.352018 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:52:37.352200 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260927 00:52:37.352509 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.342757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54777 (local address 127.0.98.62:43023)
0927 00:52:37.343295 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:37.343300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:37.343317 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:52:37.343362 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:37.343365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:37.343411 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:37.343521 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:37.344186 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.344934 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.346169 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.346541 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.346665 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.348115 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:37.348119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:37.348122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:37.348149 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:37.349667 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.350827 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.350832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.350835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.350967 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.351432 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.351435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.351436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.351676 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:37.351780 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:37.352127 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 00:52:37.352343 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 00:52:37.663923 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.328s user 0.002s sys 0.002s
I20260927 00:52:37.664435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43023
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:37.781723 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:37.781991 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:37.782047 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:37.786154 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:37.786230 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:37.786271 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:52:37.786284 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:52:37.786316 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:37.791744 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:37.792793 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:37.793782 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:37.804049 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:37.804047 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:37.804122 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:37.804596 2341 server_base.cc:1034] running on GCE node
I20260927 00:52:37.805014 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:37.805675 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:37.806850 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790470357806814 us; error 48 us; skew 500 ppm
Sep 27 00:52:37 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:37.809664 2341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:37.810591 2341 webserver.cc:460] Webserver started at http://127.0.98.2:34495/ using document root <none> and password file <none>
I20260927 00:52:37.811201 2341 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:37.811271 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:37.811470 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:37.813531 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "dc92387901e54e7ebde44db98a36f823"
format_stamp: "Formatted at 2026-09-27 00:52:37 on dist-test-slave-9fh2"
server_key: "4c19c64152aa48b7529f48564159a639"
server_key_iv: "8051854e43ad494751776de659f0b599"
server_key_version: "encryptionkey@0"
I20260927 00:52:37.814056 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dc92387901e54e7ebde44db98a36f823"
format_stamp: "Formatted at 2026-09-27 00:52:37 on dist-test-slave-9fh2"
server_key: "4c19c64152aa48b7529f48564159a639"
server_key_iv: "8051854e43ad494751776de659f0b599"
server_key_version: "encryptionkey@0"
I20260927 00:52:37.818089 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:52:37.820879 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:37.822434 2341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:52:37.822567 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "dc92387901e54e7ebde44db98a36f823"
format_stamp: "Formatted at 2026-09-27 00:52:37 on dist-test-slave-9fh2"
server_key: "4c19c64152aa48b7529f48564159a639"
server_key_iv: "8051854e43ad494751776de659f0b599"
server_key_version: "encryptionkey@0"
I20260927 00:52:37.822685 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:37.835709 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:37.839010 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:37.839241 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:37.840775 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:37.840829 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:37.840883 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:37.840909 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:37.851454 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43759
I20260927 00:52:37.851627 2471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43759 every 8 connection(s)
Sep 27 00:52:37 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:37.852849 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:37.857029 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.842267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52975 (local address 127.0.98.62:43023)
0927 00:52:37.842488 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:37.842492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:37.843638 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 00:52:37.845025 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:37.845029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:37.845101 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:37.845158 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:37.846995 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.847752 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.849271 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.849598 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.849720 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.853217 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:37.853222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:37.853225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:37.853272 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:37.854814 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.855429 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.855432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.855433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.855535 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.855856 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.855858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.855860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.856076 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:37.856165 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:37.856726 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 00:52:37.856864 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 00:52:37.857761 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.842764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.98.2:52975)
0927 00:52:37.843434 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:37.843503 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:37.844768 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:37.845615 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:37.845624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:37.846030 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:37.846787 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.846813 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.847903 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.847906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.849136 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.849147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.849826 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.849834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.850111 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.850884 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:37.850905 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:37.852959 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:37.854935 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.854940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.854952 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.855312 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.855638 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.855639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.855641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.855765 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.856193 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:37.856198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:37.856447 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 00:52:37.856631 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:52:37.856946 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260927 00:52:37.859103 2472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43023
I20260927 00:52:37.859341 2472 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:37.860061 2472 heartbeater.cc:511] Master 127.0.98.62:43023 requested a full tablet report, sending...
I20260927 00:52:37.861197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 2341
I20260927 00:52:37.861325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:37.861666 392 external_mini_cluster.cc:1428] Setting key 6633ec6b7880629d78b5627c6b738c13
I20260927 00:52:37.862031 2129 ts_manager.cc:194] Registered new tserver with Master: dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759)
I20260927 00:52:37.862833 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52975
I20260927 00:52:37.869498 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.862452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40097 (local address 127.0.98.62:43023)
0927 00:52:37.862579 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:37.862584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:37.862960 (+ 376us) server_negotiation.cc:407] Connection header received
0927 00:52:37.862997 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:37.863000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:37.863051 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:37.863128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:37.863651 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.864392 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.865243 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.865555 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.865668 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.866787 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:37.866791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:37.866794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:37.866816 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:37.868025 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.868396 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.868400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.868401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.868465 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:37.868700 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:37.868703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:37.868705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:37.868902 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:37.868986 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:37.869236 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:52:37.869341 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:52:37.869509 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:37.862366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.98.2:40097)
0927 00:52:37.862728 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:37.862740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:37.862846 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:37.863220 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:37.863224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:37.863400 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:37.863519 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.863524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.864513 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.864516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.865129 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:37.865137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:37.865783 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:37.865786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:37.865858 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:37.866318 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:37.866329 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:37.866659 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:37.868126 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.868129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.868132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.868313 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.868542 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:37.868545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:37.868546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:37.868618 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:37.868988 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:37.868991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:37.869091 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:52:37.869175 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:52:37.869324 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:37.890774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43023
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:38.022428 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:38.022805 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:38.022871 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:38.027458 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:38.027546 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:38.027597 2481 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:52:38.027616 2481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:52:38.027678 2481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:38.034098 2481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:38.035655 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:38.037029 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:38.048679 2489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:38.048730 2486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:38.048777 2487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:38.049222 2481 server_base.cc:1034] running on GCE node
I20260927 00:52:38.049697 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:38.050469 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:38.051651 2481 hybrid_clock.cc:648] HybridClock initialized: now 1790470358051608 us; error 56 us; skew 500 ppm
Sep 27 00:52:38 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:38.054935 2481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:38.056223 2481 webserver.cc:460] Webserver started at http://127.0.98.3:36727/ using document root <none> and password file <none>
I20260927 00:52:38.056942 2481 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:38.057036 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:38.057278 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:38.059923 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "639c00e62d344b49bcd8121cb520dcee"
format_stamp: "Formatted at 2026-09-27 00:52:38 on dist-test-slave-9fh2"
server_key: "6448e363f429c7a98872baff83ec956e"
server_key_iv: "af42f65cadbbe830f326395bc15f4922"
server_key_version: "encryptionkey@0"
I20260927 00:52:38.060567 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "639c00e62d344b49bcd8121cb520dcee"
format_stamp: "Formatted at 2026-09-27 00:52:38 on dist-test-slave-9fh2"
server_key: "6448e363f429c7a98872baff83ec956e"
server_key_iv: "af42f65cadbbe830f326395bc15f4922"
server_key_version: "encryptionkey@0"
I20260927 00:52:38.065339 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:52:38.068753 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:38.070549 2481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:52:38.070716 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "639c00e62d344b49bcd8121cb520dcee"
format_stamp: "Formatted at 2026-09-27 00:52:38 on dist-test-slave-9fh2"
server_key: "6448e363f429c7a98872baff83ec956e"
server_key_iv: "af42f65cadbbe830f326395bc15f4922"
server_key_version: "encryptionkey@0"
I20260927 00:52:38.070866 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:38.103390 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:38.107344 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:38.107558 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:38.109426 2481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:38.109479 2481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:38.109535 2481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:38.109563 2481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:52:38 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:38.121757 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39943
I20260927 00:52:38.122917 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:38.123198 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39943 every 8 connection(s)
I20260927 00:52:38.126253 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.110576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43595 (local address 127.0.98.62:43023)
0927 00:52:38.110706 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.110710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.112085 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 00:52:38.113478 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.113481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.113538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.113621 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:38.115705 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.116249 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.118174 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.118456 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.118547 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.121952 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:38.121956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:38.121959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:38.121990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:38.123793 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.124464 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.124469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.124470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.124588 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.124968 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.124971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.124972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.125177 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:38.125261 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.125959 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.126109 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:52:38.127319 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.111286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.98.3:43595)
0927 00:52:38.111910 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.111968 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.113247 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.113946 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.113957 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.114522 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:38.115456 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.115481 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.116383 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.116387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.118009 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.118024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.118626 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.118630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.118902 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.119748 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:38.119775 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:38.121709 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:38.123934 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:38.123971 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:38.123987 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:38.124301 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:38.124712 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:38.124715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:38.124717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:38.124871 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:38.125297 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:38.125302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:38.125613 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.125831 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.126200 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":162,"threads_started":1}
I20260927 00:52:38.128775 2612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43023
I20260927 00:52:38.128990 2612 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:38.129245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 2481
I20260927 00:52:38.129354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:38.129690 392 external_mini_cluster.cc:1428] Setting key 4e62c949de03ed83a25890d5a9c6bf44
I20260927 00:52:38.129787 2612 heartbeater.cc:511] Master 127.0.98.62:43023 requested a full tablet report, sending...
I20260927 00:52:38.131114 2128 ts_manager.cc:194] Registered new tserver with Master: 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:38.131927 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43595
I20260927 00:52:38.132969 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:52:38.139508 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.131405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.98.3:44195)
0927 00:52:38.131669 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.131682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.131768 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.132005 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.132008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.132226 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:38.132342 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.132348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.133435 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.133440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.134299 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.134312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.134761 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.134765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.134870 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.135591 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:38.135603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:38.136012 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:38.137826 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:38.137830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:38.137832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:38.138092 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:38.138433 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:38.138436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:38.138438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:38.138542 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:38.138934 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:38.138938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:38.139027 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.139114 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.139318 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260927 00:52:38.139513 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.131373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44195 (local address 127.0.98.62:43023)
0927 00:52:38.131508 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.131514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.131750 (+ 236us) server_negotiation.cc:407] Connection header received
0927 00:52:38.131840 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.131843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.131889 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.131975 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:38.132458 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.133274 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.134432 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.134651 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.134735 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.136178 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:38.136182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:38.136184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:38.136213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:38.137710 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.138212 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.138215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.138216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.138300 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.138660 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.138663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.138665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.138845 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:38.138932 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.139204 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.139312 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:52:38 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:52:38 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:38.162464 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.153644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.0.98.62:43023)
0927 00:52:38.153855 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.153860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.153885 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:52:38.154063 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.154067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.154119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.154191 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:38.154980 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.155611 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.156458 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.156693 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.156790 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.159425 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:38.159428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:38.159431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:38.159457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:38.160921 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.161319 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.161323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.161326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.161405 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.161637 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.161641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.161643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.161835 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:38.161954 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.162162 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.162285 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:52:38.166155 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:38.170087 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:38.187330 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.181609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.98.1:39385)
0927 00:52:38.182058 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.182066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.182469 (+ 403us) server_negotiation.cc:407] Connection header received
0927 00:52:38.182562 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.182567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.182787 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.182942 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.183400 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.184208 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.185941 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.186572 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.186677 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.186863 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.187003 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.187118 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":81,"threads_started":1}
I20260927 00:52:38.187330 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.181044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.0.1:33588)
0927 00:52:38.182239 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.182252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.182376 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.183117 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.183120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.183139 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.183275 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.183281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.184317 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.184321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.185792 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.185805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.186692 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.186696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.186808 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.186847 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.186964 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.187114 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":1,"thread_start_us":133,"threads_started":1}
I20260927 00:52:38.187579 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.181998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.0.1:44302)
0927 00:52:38.182494 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.182507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.182611 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.183345 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.183348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.183365 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.183443 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.183448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.184763 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.184766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.185936 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.185946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.187033 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.187037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.187187 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.187250 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.187313 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.187403 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":69,"thread_start_us":112,"threads_started":1}
I20260927 00:52:38.187765 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.180975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43759 (local address 127.0.0.1:46472)
0927 00:52:38.181504 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.181561 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.181709 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.182801 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.182808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.182831 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.183022 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.183029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.184639 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.184643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.185889 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.185896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.187033 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.187037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.187192 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.187498 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.187570 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.187655 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":289,"thread_start_us":198,"threads_started":1}
I20260927 00:52:38.187955 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.181102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.98.2:43759)
0927 00:52:38.182287 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.182300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.182324 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:52:38.182405 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.182411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.182632 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.183218 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.183350 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.184522 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.186051 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.186895 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.187268 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.187446 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.187660 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.187798 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":123,"threads_started":1}
I20260927 00:52:38.188308 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.182209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.0.98.3:39943)
0927 00:52:38.182894 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.182905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.182929 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:52:38.182998 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.183003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.183210 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.183349 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.183554 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.184641 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.186069 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.186895 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.187856 (+ 961us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.188018 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.188072 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.188180 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":134,"threads_started":1}
I20260927 00:52:38.190860 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9586a8cb4c409f9cf8f230c722ad31 (DEFAULT_TABLE table=test-table [id=05e3307458a74cd6a273965b8ab89a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:38.191404 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9586a8cb4c409f9cf8f230c722ad31 (DEFAULT_TABLE table=test-table [id=05e3307458a74cd6a273965b8ab89a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:38.191404 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9586a8cb4c409f9cf8f230c722ad31 (DEFAULT_TABLE table=test-table [id=05e3307458a74cd6a273965b8ab89a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:38.191922 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9586a8cb4c409f9cf8f230c722ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.192634 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9586a8cb4c409f9cf8f230c722ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.192636 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9586a8cb4c409f9cf8f230c722ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.198328 2636 tablet_bootstrap.cc:492] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Bootstrap starting.
I20260927 00:52:38.200395 2636 tablet_bootstrap.cc:654] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.201189 2636 log.cc:824] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:38.202322 2639 tablet_bootstrap.cc:492] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Bootstrap starting.
I20260927 00:52:38.202920 2637 tablet_bootstrap.cc:492] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Bootstrap starting.
I20260927 00:52:38.203385 2636 tablet_bootstrap.cc:492] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: No bootstrap required, opened a new log
I20260927 00:52:38.203610 2636 ts_tablet_manager.cc:1397] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 00:52:38.204886 2639 tablet_bootstrap.cc:654] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.205142 2637 tablet_bootstrap.cc:654] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.205720 2639 log.cc:824] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:38.206118 2637 log.cc:824] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:38.207875 2636 raft_consensus.cc:348] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.208185 2636 raft_consensus.cc:374] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.208244 2636 raft_consensus.cc:729] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.208276 2639 tablet_bootstrap.cc:492] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: No bootstrap required, opened a new log
I20260927 00:52:38.208513 2639 ts_tablet_manager.cc:1397] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:52:38.208782 2637 tablet_bootstrap.cc:492] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: No bootstrap required, opened a new log
I20260927 00:52:38.208700 2636 consensus_queue.cc:260] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.209013 2637 ts_tablet_manager.cc:1397] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:52:38.209827 2636 ts_tablet_manager.cc:1428] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:52:38.211419 2332 heartbeater.cc:503] Master 127.0.98.62:43023 was elected leader, sending a full tablet report...
I20260927 00:52:38.212627 2637 raft_consensus.cc:348] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.212910 2637 raft_consensus.cc:374] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.212972 2637 raft_consensus.cc:729] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc92387901e54e7ebde44db98a36f823, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.212739 2639 raft_consensus.cc:348] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.213049 2639 raft_consensus.cc:374] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.213111 2639 raft_consensus.cc:729] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.213442 2637 consensus_queue.cc:260] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.213644 2639 consensus_queue.cc:260] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.214340 2472 heartbeater.cc:503] Master 127.0.98.62:43023 was elected leader, sending a full tablet report...
I20260927 00:52:38.214586 2637 ts_tablet_manager.cc:1428] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:52:38.215312 2639 ts_tablet_manager.cc:1428] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:52:38.215694 2612 heartbeater.cc:503] Master 127.0.98.62:43023 was elected leader, sending a full tablet report...
I20260927 00:52:38.215930 2643 raft_consensus.cc:482] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:38.216086 2643 raft_consensus.cc:504] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.217270 2643 leader_election.cc:288] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385), 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:38.221772 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.217686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33525 (local address 127.0.98.1:39385)
0927 00:52:38.217836 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.217842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.217994 (+ 152us) server_negotiation.cc:407] Connection header received
0927 00:52:38.218133 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.218136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.218190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.218284 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.218698 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.219423 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.220828 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.221252 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.221342 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.221406 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.221566 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.221644 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:52:38.221772 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.217503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.98.2:33525)
0927 00:52:38.217893 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.217910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.218013 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.218423 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.218428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.218452 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.218572 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.218580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.219552 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.219555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.220721 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.220730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.221358 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.221361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.221462 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.221480 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.221543 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.221631 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260927 00:52:38.222419 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.217544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.98.2:57081)
0927 00:52:38.218268 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.218281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.218376 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.218597 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.218600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.218616 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.218765 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.218771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.219789 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.219792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.221265 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.221276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.222028 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.222032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.222129 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.222152 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.222225 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.222297 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":153,"threads_started":1}
I20260927 00:52:38.222451 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.217617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57081 (local address 127.0.98.3:39943)
0927 00:52:38.217743 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.217748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.218350 (+ 602us) server_negotiation.cc:407] Connection header received
0927 00:52:38.218460 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.218464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.218513 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.218576 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.218881 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.219659 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.221384 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.221812 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.221959 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.222049 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.222247 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.222350 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:52:38.222536 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9586a8cb4c409f9cf8f230c722ad31" candidate_uuid: "dc92387901e54e7ebde44db98a36f823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" is_pre_election: true
I20260927 00:52:38.222985 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9586a8cb4c409f9cf8f230c722ad31" candidate_uuid: "dc92387901e54e7ebde44db98a36f823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639c00e62d344b49bcd8121cb520dcee" is_pre_election: true
I20260927 00:52:38.223016 2286 raft_consensus.cc:2454] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc92387901e54e7ebde44db98a36f823 in term 0.
I20260927 00:52:38.223307 2566 raft_consensus.cc:2454] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc92387901e54e7ebde44db98a36f823 in term 0.
I20260927 00:52:38.223652 2360 leader_election.cc:302] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce4c5f0c7e1e40c3bad55067b70e55b7, dc92387901e54e7ebde44db98a36f823; no voters:
I20260927 00:52:38.224001 2643 raft_consensus.cc:2781] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:38.224084 2643 raft_consensus.cc:482] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:38.224134 2643 raft_consensus.cc:3037] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.225235 2643 raft_consensus.cc:504] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.225582 2643 leader_election.cc:288] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [CANDIDATE]: Term 1 election: Requested vote from peers ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385), 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:38.225939 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9586a8cb4c409f9cf8f230c722ad31" candidate_uuid: "dc92387901e54e7ebde44db98a36f823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639c00e62d344b49bcd8121cb520dcee"
I20260927 00:52:38.226045 2566 raft_consensus.cc:3037] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.226127 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9586a8cb4c409f9cf8f230c722ad31" candidate_uuid: "dc92387901e54e7ebde44db98a36f823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7"
I20260927 00:52:38.226301 2286 raft_consensus.cc:3037] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.227293 2566 raft_consensus.cc:2454] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc92387901e54e7ebde44db98a36f823 in term 1.
I20260927 00:52:38.227618 2357 leader_election.cc:302] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, dc92387901e54e7ebde44db98a36f823; no voters:
I20260927 00:52:38.227859 2286 raft_consensus.cc:2454] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc92387901e54e7ebde44db98a36f823 in term 1.
I20260927 00:52:38.227882 2643 raft_consensus.cc:2781] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:38.228135 2643 raft_consensus.cc:686] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 1 LEADER]: Becoming Leader. State: Replica: dc92387901e54e7ebde44db98a36f823, State: Running, Role: LEADER
I20260927 00:52:38.228653 2643 consensus_queue.cc:237] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.232764 2129 catalog_manager.cc:5495] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc92387901e54e7ebde44db98a36f823 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "dc92387901e54e7ebde44db98a36f823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:52:38 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:38.283030 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.273344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43023 (local address 127.0.0.1:57098)
0927 00:52:38.273776 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.273791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.273924 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.274197 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.274200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.274716 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:38.274831 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.274839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.275595 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.275599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.276238 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.276247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.276655 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.276658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.276786 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.277438 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:38.277454 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:38.279291 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:38.281056 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:38.281068 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:38.281075 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:38.281382 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:38.281699 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:38.281702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:38.281704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:38.281828 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:38.282285 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:38.282292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:38.282511 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.282633 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.282850 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":33,"thread_start_us":173,"threads_started":1}
I20260927 00:52:38.283126 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.273457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.0.98.62:43023)
0927 00:52:38.273650 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.273655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.273883 (+ 228us) server_negotiation.cc:407] Connection header received
0927 00:52:38.274060 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.274062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.274098 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.274163 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:38.274937 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.275501 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.276349 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.276558 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.276646 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.279489 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:38.279492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:38.279495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:38.279518 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:38.280932 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.281499 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.281502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.281504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.281584 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:38.281928 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:38.281931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:38.281932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:38.282114 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:38.282236 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.282735 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.282929 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":88}
I20260927 00:52:38.287847 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57098:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:52:38.288635 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:38.294303 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 722e84ad50204a0f9a3d69515ac9e0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99d1d36d384a4284869925922fcca247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:52:38.294643 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 722e84ad50204a0f9a3d69515ac9e0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99d1d36d384a4284869925922fcca247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:52:38.294505 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 722e84ad50204a0f9a3d69515ac9e0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99d1d36d384a4284869925922fcca247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:52:38.295213 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722e84ad50204a0f9a3d69515ac9e0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.295231 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722e84ad50204a0f9a3d69515ac9e0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.295416 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722e84ad50204a0f9a3d69515ac9e0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.298305 2636 tablet_bootstrap.cc:492] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: Bootstrap starting.
I20260927 00:52:38.298393 2639 tablet_bootstrap.cc:492] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: Bootstrap starting.
I20260927 00:52:38.299306 2637 tablet_bootstrap.cc:492] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: Bootstrap starting.
I20260927 00:52:38.299516 2636 tablet_bootstrap.cc:654] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.300146 2639 tablet_bootstrap.cc:654] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.300277 2637 tablet_bootstrap.cc:654] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.301052 2636 tablet_bootstrap.cc:492] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: No bootstrap required, opened a new log
I20260927 00:52:38.301144 2636 ts_tablet_manager.cc:1397] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:38.301556 2639 tablet_bootstrap.cc:492] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: No bootstrap required, opened a new log
I20260927 00:52:38.301666 2639 ts_tablet_manager.cc:1397] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:38.301649 2636 raft_consensus.cc:348] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.301771 2636 raft_consensus.cc:374] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.301802 2636 raft_consensus.cc:729] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.301987 2637 tablet_bootstrap.cc:492] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: No bootstrap required, opened a new log
I20260927 00:52:38.302078 2637 ts_tablet_manager.cc:1397] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:38.302161 2636 consensus_queue.cc:260] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.302196 2639 raft_consensus.cc:348] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.302330 2639 raft_consensus.cc:374] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.302361 2639 raft_consensus.cc:729] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.302438 2636 ts_tablet_manager.cc:1428] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:52:38.302518 2642 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.302600 2637 raft_consensus.cc:348] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.302707 2642 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.302721 2637 raft_consensus.cc:374] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.302752 2637 raft_consensus.cc:729] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc92387901e54e7ebde44db98a36f823, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.302881 2637 consensus_queue.cc:260] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.303172 2643 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.303347 2643 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.303166 2637 ts_tablet_manager.cc:1428] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:52:38.303629 2639 consensus_queue.cc:260] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.303946 2644 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.304103 2644 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.304356 2639 ts_tablet_manager.cc:1428] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260927 00:52:38.335731 2333 tablet.cc:2369] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:52:38.354841 2473 tablet.cc:2369] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:52:38.374275 2613 tablet.cc:2369] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:52:38.584196 2644 raft_consensus.cc:482] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:38.584404 2644 raft_consensus.cc:504] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.585846 2644 leader_election.cc:288] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385), dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759)
I20260927 00:52:38.589787 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.586147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.98.3:49551)
0927 00:52:38.586345 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.586364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.586484 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.586901 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.586904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.586918 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.587081 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.587087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.587856 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.587859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.588696 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.588705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.589348 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.589351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.589448 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.589468 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.589532 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.589620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":32}
I20260927 00:52:38.589888 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.586423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56567 (local address 127.0.98.2:43759)
0927 00:52:38.586535 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.586540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.586648 (+ 108us) server_negotiation.cc:407] Connection header received
0927 00:52:38.586804 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.586807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.586851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.586914 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.587323 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.587835 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.588922 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.589330 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.589421 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.589483 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.589693 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.589759 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:52:38.590260 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.586189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43759 (local address 127.0.98.3:56567)
0927 00:52:38.586528 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.586544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.586701 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.586939 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.586942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.586958 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:38.587215 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.587224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.587942 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.587945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.588801 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.588810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.589462 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.589465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.589576 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.589598 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.589664 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.589745 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":25,"thread_start_us":125,"threads_started":1}
I20260927 00:52:38.590408 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.586231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49551 (local address 127.0.98.1:39385)
0927 00:52:38.586366 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.586370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.586581 (+ 211us) server_negotiation.cc:407] Connection header received
0927 00:52:38.586652 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.586655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.586707 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.586772 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:38.587206 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.587748 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.588824 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.589236 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.589334 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.589408 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.589555 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.589636 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:52:38.590919 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722e84ad50204a0f9a3d69515ac9e0b9" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" is_pre_election: true
I20260927 00:52:38.591079 2286 raft_consensus.cc:2454] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 0.
I20260927 00:52:38.591048 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722e84ad50204a0f9a3d69515ac9e0b9" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823" is_pre_election: true
I20260927 00:52:38.591331 2424 raft_consensus.cc:2454] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 0.
I20260927 00:52:38.591585 2500 leader_election.cc:302] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, ce4c5f0c7e1e40c3bad55067b70e55b7; no voters:
I20260927 00:52:38.591940 2644 raft_consensus.cc:2781] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:38.592005 2644 raft_consensus.cc:482] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:38.592058 2644 raft_consensus.cc:3037] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.593109 2644 raft_consensus.cc:504] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.593499 2644 leader_election.cc:288] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 election: Requested vote from peers ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385), dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759)
I20260927 00:52:38.593950 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722e84ad50204a0f9a3d69515ac9e0b9" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7"
I20260927 00:52:38.593990 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722e84ad50204a0f9a3d69515ac9e0b9" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823"
I20260927 00:52:38.594098 2424 raft_consensus.cc:3037] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.594110 2286 raft_consensus.cc:3037] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.595193 2424 raft_consensus.cc:2454] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 1.
I20260927 00:52:38.595381 2286 raft_consensus.cc:2454] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 1.
I20260927 00:52:38.595546 2499 leader_election.cc:302] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, dc92387901e54e7ebde44db98a36f823; no voters:
I20260927 00:52:38.595810 2644 raft_consensus.cc:2781] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:38.595988 2644 raft_consensus.cc:686] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [term 1 LEADER]: Becoming Leader. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Running, Role: LEADER
I20260927 00:52:38.596357 2644 consensus_queue.cc:237] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.598357 2658 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: TxnStatusTablet state changed. Reason: New leader 639c00e62d344b49bcd8121cb520dcee. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.598505 2658 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:52:38.598841 2660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:52:38.599066 2660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:52:38.599726 2128 catalog_manager.cc:5495] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee reported cstate change: term changed from 0 to 1, leader changed from <none> to 639c00e62d344b49bcd8121cb520dcee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } health_report { overall_health: HEALTHY } } }
I20260927 00:52:38.676293 2643 consensus_queue.cc:1035] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:38.677150 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.672579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.0.1:44310)
0927 00:52:38.672757 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.672774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.672884 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.673228 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.673231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.673238 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:38.673333 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.673337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.674037 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.674040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.674867 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.674879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.675611 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.675615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.675700 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.675761 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.676797 (+ 1036us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.676814 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.676890 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.677000 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:52:38.677201 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.672840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.0.98.3:39943)
0927 00:52:38.672974 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.672978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.672991 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:52:38.673064 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.673067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.673118 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.673194 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:38.673423 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.673936 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.675030 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.675463 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.675583 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.675989 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.676698 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.676824 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.676914 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.677017 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:52:38.683040 2666 consensus_queue.cc:1035] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:38.683342 2424 raft_consensus.cc:1261] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Refusing update from remote peer 639c00e62d344b49bcd8121cb520dcee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:38.683947 2644 consensus_queue.cc:1035] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:38.684327 2285 raft_consensus.cc:1261] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Refusing update from remote peer 639c00e62d344b49bcd8121cb520dcee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:38.685058 2658 consensus_queue.cc:1035] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:38.687507 2666 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: TxnStatusTablet state changed. Reason: New leader 639c00e62d344b49bcd8121cb520dcee. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.687649 2666 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.689404 2642 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: TxnStatusTablet state changed. Reason: New leader 639c00e62d344b49bcd8121cb520dcee. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.689570 2642 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.693321 2644 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.693477 2644 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:52:38.693823 2642 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.693953 2642 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P ce4c5f0c7e1e40c3bad55067b70e55b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.696216 2668 mvcc.cc:205] Tried to move back new op lower bound from 7333766589159194624 to 7333766588813447168. Current Snapshot: MvccSnapshot[applied={T|T < 7333766589159194624}]
I20260927 00:52:38.700491 2644 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.700649 2644 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P 639c00e62d344b49bcd8121cb520dcee: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:52:38.702404 2666 tablet_replica.cc:439] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } } }
I20260927 00:52:38.702538 2666 tablet_replica.cc:442] T 722e84ad50204a0f9a3d69515ac9e0b9 P dc92387901e54e7ebde44db98a36f823: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:52:38.702873 2665 mvcc.cc:205] Tried to move back new op lower bound from 7333766589159194624 to 7333766588813447168. Current Snapshot: MvccSnapshot[applied={T|T < 7333766589159194624}]
I20260927 00:52:38.709108 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.705291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.0.98.2:43759)
0927 00:52:38.705395 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.705398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.705743 (+ 345us) server_negotiation.cc:407] Connection header received
0927 00:52:38.705900 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.705904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.705956 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.706040 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:38.706279 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.706779 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.707728 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.707951 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.708034 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.708291 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.708756 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.708852 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.708923 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.708975 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:52:38.721518 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.718230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58403 (local address 127.0.98.3:39943)
0927 00:52:38.718351 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.718356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.718585 (+ 229us) server_negotiation.cc:407] Connection header received
0927 00:52:38.718747 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.718750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.718795 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.718855 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:38.719172 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.719673 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.720499 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.720705 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.720787 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.721096 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.721169 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.721226 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.721318 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.721371 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:52:38.721529 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.718154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.98.2:58403)
0927 00:52:38.718497 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.718511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.718648 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.718872 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.718875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.718882 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:38.718997 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.719002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.719773 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.719777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.720394 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.720401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.720809 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.720812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.720899 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.720943 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.721240 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.721247 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.721300 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.721372 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":141,"threads_started":1}
I20260927 00:52:38.752995 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.747347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47151 (local address 127.0.98.2:43759)
0927 00:52:38.747479 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.747482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.747825 (+ 343us) server_negotiation.cc:407] Connection header received
0927 00:52:38.747978 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.747981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.748038 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.748132 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:38.748738 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.749363 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.750771 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.751083 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.751225 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.752143 (+ 918us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.752235 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.752309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.752774 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.752842 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:52:38.753116 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.747313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43759 (local address 127.0.98.3:47151)
0927 00:52:38.747743 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.747761 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.747878 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.748176 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.748180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.748192 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:38.748386 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.748393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.749566 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.749570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.750596 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.750609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.751227 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.751231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.751339 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.751573 (+ 234us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.752616 (+ 1043us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.752623 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.752736 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.752859 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":166,"threads_started":1}
I20260927 00:52:38.780694 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:38.780882 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:38.783844 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Sending DeleteTablet for 3 replicas of tablet 5c9586a8cb4c409f9cf8f230c722ad31
I20260927 00:52:38.784641 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9586a8cb4c409f9cf8f230c722ad31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46472
I20260927 00:52:38.784709 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9586a8cb4c409f9cf8f230c722ad31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33588
I20260927 00:52:38.785331 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9586a8cb4c409f9cf8f230c722ad31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260927 00:52:38.785840 2685 tablet_replica.cc:330] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: stopping tablet replica
I20260927 00:52:38.785885 2686 tablet_replica.cc:330] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: stopping tablet replica
I20260927 00:52:38.786023 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:52:38.786203 2685 raft_consensus.cc:2229] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:38.786439 2685 raft_consensus.cc:2258] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:52:38.786476 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:38.786711 2686 raft_consensus.cc:2229] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:38.787021 2686 raft_consensus.cc:2258] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:38.787919 2685 ts_tablet_manager.cc:1905] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:38.788689 2686 ts_tablet_manager.cc:1905] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:38.791276 2685 ts_tablet_manager.cc:1918] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:38.791389 2685 log.cc:1196] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/5c9586a8cb4c409f9cf8f230c722ad31
I20260927 00:52:38.791715 2685 ts_tablet_manager.cc:1939] T 5c9586a8cb4c409f9cf8f230c722ad31 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting consensus metadata
I20260927 00:52:38.791927 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7ed7d813a7461986c5d097cea7a828 (DEFAULT_TABLE table=test-table [id=08ac0ff8a2aa4a8c9faaf55e3559ccdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:38.792172 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7ed7d813a7461986c5d097cea7a828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.792802 2686 ts_tablet_manager.cc:1918] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:38.792979 2686 log.cc:1196] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/5c9586a8cb4c409f9cf8f230c722ad31
I20260927 00:52:38.793118 2684 tablet_replica.cc:330] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: stopping tablet replica
I20260927 00:52:38.793396 2686 ts_tablet_manager.cc:1939] T 5c9586a8cb4c409f9cf8f230c722ad31 P 639c00e62d344b49bcd8121cb520dcee: Deleting consensus metadata
I20260927 00:52:38.793479 2684 raft_consensus.cc:2229] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:38.793866 2684 raft_consensus.cc:2258] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:38.794317 2636 tablet_bootstrap.cc:492] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Bootstrap starting.
I20260927 00:52:38.794482 2116 catalog_manager.cc:4841] TS ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385): tablet 5c9586a8cb4c409f9cf8f230c722ad31 (table test-table [id=05e3307458a74cd6a273965b8ab89a75]) successfully deleted
I20260927 00:52:38.794987 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7ed7d813a7461986c5d097cea7a828 (DEFAULT_TABLE table=test-table [id=08ac0ff8a2aa4a8c9faaf55e3559ccdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:38.795275 2636 tablet_bootstrap.cc:654] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.795347 2113 catalog_manager.cc:4841] TS 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943): tablet 5c9586a8cb4c409f9cf8f230c722ad31 (table test-table [id=05e3307458a74cd6a273965b8ab89a75]) successfully deleted
I20260927 00:52:38.795600 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7ed7d813a7461986c5d097cea7a828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.795814 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7ed7d813a7461986c5d097cea7a828 (DEFAULT_TABLE table=test-table [id=08ac0ff8a2aa4a8c9faaf55e3559ccdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:38.796089 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7ed7d813a7461986c5d097cea7a828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:38.796837 2684 ts_tablet_manager.cc:1905] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:38.798007 2637 tablet_bootstrap.cc:492] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Bootstrap starting.
I20260927 00:52:38.798681 2639 tablet_bootstrap.cc:492] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Bootstrap starting.
I20260927 00:52:38.798946 2637 tablet_bootstrap.cc:654] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.799558 2684 ts_tablet_manager.cc:1918] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:38.799631 2639 tablet_bootstrap.cc:654] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:38.799640 2684 log.cc:1196] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/5c9586a8cb4c409f9cf8f230c722ad31
I20260927 00:52:38.799927 2684 ts_tablet_manager.cc:1939] T 5c9586a8cb4c409f9cf8f230c722ad31 P dc92387901e54e7ebde44db98a36f823: Deleting consensus metadata
I20260927 00:52:38.801149 2636 tablet_bootstrap.cc:492] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: No bootstrap required, opened a new log
I20260927 00:52:38.801258 2636 ts_tablet_manager.cc:1397] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260927 00:52:38.801719 2636 raft_consensus.cc:348] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.801836 2636 raft_consensus.cc:374] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.801877 2636 raft_consensus.cc:729] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.801991 2636 consensus_queue.cc:260] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.802199 2637 tablet_bootstrap.cc:492] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: No bootstrap required, opened a new log
I20260927 00:52:38.802284 2637 ts_tablet_manager.cc:1397] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 00:52:38.802289 2636 ts_tablet_manager.cc:1428] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:52:38.802706 2637 raft_consensus.cc:348] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.802815 2637 raft_consensus.cc:374] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.802819 2639 tablet_bootstrap.cc:492] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: No bootstrap required, opened a new log
I20260927 00:52:38.802847 2637 raft_consensus.cc:729] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc92387901e54e7ebde44db98a36f823, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.802861 2115 catalog_manager.cc:4841] TS dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759): tablet 5c9586a8cb4c409f9cf8f230c722ad31 (table test-table [id=05e3307458a74cd6a273965b8ab89a75]) successfully deleted
I20260927 00:52:38.802906 2639 ts_tablet_manager.cc:1397] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:52:38.802963 2637 consensus_queue.cc:260] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.803252 2637 ts_tablet_manager.cc:1428] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:52:38.803369 2639 raft_consensus.cc:348] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.803485 2639 raft_consensus.cc:374] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:38.803512 2639 raft_consensus.cc:729] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Initialized, Role: FOLLOWER
I20260927 00:52:38.803644 2639 consensus_queue.cc:260] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.803929 2639 ts_tablet_manager.cc:1428] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:52:38.887809 2673 raft_consensus.cc:482] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:38.887959 2673 raft_consensus.cc:504] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.888412 2673 leader_election.cc:288] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385), dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759)
I20260927 00:52:38.888813 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7ed7d813a7461986c5d097cea7a828" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" is_pre_election: true
I20260927 00:52:38.888959 2285 raft_consensus.cc:2454] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 0.
I20260927 00:52:38.889010 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7ed7d813a7461986c5d097cea7a828" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823" is_pre_election: true
I20260927 00:52:38.889240 2424 raft_consensus.cc:2454] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 0.
I20260927 00:52:38.889322 2500 leader_election.cc:302] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, ce4c5f0c7e1e40c3bad55067b70e55b7; no voters:
I20260927 00:52:38.889544 2673 raft_consensus.cc:2781] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:38.889597 2673 raft_consensus.cc:482] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:38.889636 2673 raft_consensus.cc:3037] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.890674 2673 raft_consensus.cc:504] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.891002 2673 leader_election.cc:288] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 election: Requested vote from peers ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385), dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759)
I20260927 00:52:38.891446 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7ed7d813a7461986c5d097cea7a828" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823"
I20260927 00:52:38.891413 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7ed7d813a7461986c5d097cea7a828" candidate_uuid: "639c00e62d344b49bcd8121cb520dcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7"
I20260927 00:52:38.891582 2424 raft_consensus.cc:3037] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.891588 2285 raft_consensus.cc:3037] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:38.892531 2424 raft_consensus.cc:2454] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 1.
I20260927 00:52:38.892802 2499 leader_election.cc:302] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, dc92387901e54e7ebde44db98a36f823; no voters:
I20260927 00:52:38.892963 2673 raft_consensus.cc:2781] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:38.893045 2673 raft_consensus.cc:686] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 1 LEADER]: Becoming Leader. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Running, Role: LEADER
I20260927 00:52:38.893085 2285 raft_consensus.cc:2454] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c00e62d344b49bcd8121cb520dcee in term 1.
I20260927 00:52:38.893186 2673 consensus_queue.cc:237] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:38.895440 2129 catalog_manager.cc:5495] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee reported cstate change: term changed from 0 to 1, leader changed from <none> to 639c00e62d344b49bcd8121cb520dcee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "639c00e62d344b49bcd8121cb520dcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } health_report { overall_health: HEALTHY } } }
I20260927 00:52:38.919729 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.915901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.0.98.3:39943)
0927 00:52:38.916061 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.916066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.916085 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:52:38.916221 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.916224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.916291 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.916386 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:38.916596 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.917420 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.918287 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.918667 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.918800 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.919188 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.919298 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.919374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.919492 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.919556 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:52:38.928377 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.925637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34457 (local address 127.0.98.3:39943)
0927 00:52:38.925741 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:38.925746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:38.925758 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:52:38.925787 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:38.925789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:38.925822 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:38.925878 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:38.926102 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.926613 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.927436 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.927636 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.927718 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.927950 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.928014 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.928067 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:38.928178 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 00:52:38.928213 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":38}
I20260927 00:52:38.928377 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:38.925443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.98.3:34457)
0927 00:52:38.925575 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:38.925589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:38.925676 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:38.925891 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:38.925893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:38.925899 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:38.926027 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.926034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.926703 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.926706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.927342 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:38.927351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:38.927730 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:38.927733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:38.927840 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:38.927861 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:38.928079 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:38.928085 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:52:38.928142 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:52:38.928214 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":34}
I20260927 00:52:38.935369 2424 raft_consensus.cc:1261] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Refusing update from remote peer 639c00e62d344b49bcd8121cb520dcee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:38.935801 2285 raft_consensus.cc:1261] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Refusing update from remote peer 639c00e62d344b49bcd8121cb520dcee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:38.935818 2673 consensus_queue.cc:1035] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:38.937170 2673 consensus_queue.cc:1035] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:38.939702 2664 mvcc.cc:205] Tried to move back new op lower bound from 7333766590195916800 to 7333766590027804672. Current Snapshot: MvccSnapshot[applied={T|T < 7333766590195916800}]
W20260927 00:52:38.953060 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:52:40.525671 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:40.525918 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:40.529135 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Sending DeleteTablet for 3 replicas of tablet 3c7ed7d813a7461986c5d097cea7a828
I20260927 00:52:40.529906 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7ed7d813a7461986c5d097cea7a828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33588
I20260927 00:52:40.529906 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7ed7d813a7461986c5d097cea7a828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46472
I20260927 00:52:40.529969 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7ed7d813a7461986c5d097cea7a828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260927 00:52:40.530474 2717 tablet_replica.cc:330] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: stopping tablet replica
I20260927 00:52:40.530645 2717 raft_consensus.cc:2229] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:40.530797 2717 raft_consensus.cc:2258] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:40.531057 2716 tablet_replica.cc:330] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: stopping tablet replica
I20260927 00:52:40.531279 2716 raft_consensus.cc:2229] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:40.531298 2718 tablet_replica.cc:330] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: stopping tablet replica
I20260927 00:52:40.531458 2716 raft_consensus.cc:2258] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:40.531458 2718 raft_consensus.cc:2229] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:40.531808 2718 raft_consensus.cc:2258] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:40.532146 2717 ts_tablet_manager.cc:1905] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:40.532421 2718 ts_tablet_manager.cc:1905] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:40.534852 2717 ts_tablet_manager.cc:1918] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:52:40.534852 2718 ts_tablet_manager.cc:1918] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:52:40.534926 2718 log.cc:1196] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/3c7ed7d813a7461986c5d097cea7a828
I20260927 00:52:40.535023 2717 log.cc:1196] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/3c7ed7d813a7461986c5d097cea7a828
I20260927 00:52:40.535034 2716 ts_tablet_manager.cc:1905] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:40.535223 2718 ts_tablet_manager.cc:1939] T 3c7ed7d813a7461986c5d097cea7a828 P 639c00e62d344b49bcd8121cb520dcee: Deleting consensus metadata
I20260927 00:52:40.535410 2717 ts_tablet_manager.cc:1939] T 3c7ed7d813a7461986c5d097cea7a828 P dc92387901e54e7ebde44db98a36f823: Deleting consensus metadata
I20260927 00:52:40.536070 2113 catalog_manager.cc:4841] TS 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943): tablet 3c7ed7d813a7461986c5d097cea7a828 (table test-table [id=08ac0ff8a2aa4a8c9faaf55e3559ccdf]) successfully deleted
I20260927 00:52:40.536090 2115 catalog_manager.cc:4841] TS dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759): tablet 3c7ed7d813a7461986c5d097cea7a828 (table test-table [id=08ac0ff8a2aa4a8c9faaf55e3559ccdf]) successfully deleted
I20260927 00:52:40.537906 2716 ts_tablet_manager.cc:1918] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:52:40.538004 2716 log.cc:1196] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/3c7ed7d813a7461986c5d097cea7a828
I20260927 00:52:40.538255 2716 ts_tablet_manager.cc:1939] T 3c7ed7d813a7461986c5d097cea7a828 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting consensus metadata
I20260927 00:52:40.539127 2116 catalog_manager.cc:4841] TS ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385): tablet 3c7ed7d813a7461986c5d097cea7a828 (table test-table [id=08ac0ff8a2aa4a8c9faaf55e3559ccdf]) successfully deleted
Sep 27 00:52:40 dist-test-slave-9fh2 krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:52:40 dist-test-slave-9fh2 krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:40.566359 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.553222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.0.98.62:43023)
0927 00:52:40.553484 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:40.553490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:40.553507 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:52:40.553642 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:40.553645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:40.553690 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:40.553793 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:40.554552 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.555112 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.556107 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.556357 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.556455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.562786 (+ 6331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:40.562789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:40.562791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:40.562820 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:40.564532 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:40.565056 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:40.565059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:40.565060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:40.565148 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:40.565479 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:40.565488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:40.565489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:40.565692 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:40.565802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:40.566100 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:52:40.566214 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260927 00:52:40.569511 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:40.569926 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:40.575038 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 13262cee93a5422a8d5b6d92bf1289a5 (DEFAULT_TABLE table=test-table [id=0c7855fb79c44e4c805d32b33fc9e968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:40.575193 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 13262cee93a5422a8d5b6d92bf1289a5 (DEFAULT_TABLE table=test-table [id=0c7855fb79c44e4c805d32b33fc9e968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:40.575404 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13262cee93a5422a8d5b6d92bf1289a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:40.575203 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 13262cee93a5422a8d5b6d92bf1289a5 (DEFAULT_TABLE table=test-table [id=0c7855fb79c44e4c805d32b33fc9e968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:40.575465 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13262cee93a5422a8d5b6d92bf1289a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:40.575696 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13262cee93a5422a8d5b6d92bf1289a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:40.578156 2727 tablet_bootstrap.cc:492] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Bootstrap starting.
I20260927 00:52:40.578723 2729 tablet_bootstrap.cc:492] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Bootstrap starting.
I20260927 00:52:40.579324 2727 tablet_bootstrap.cc:654] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:40.579713 2729 tablet_bootstrap.cc:654] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:40.580027 2728 tablet_bootstrap.cc:492] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Bootstrap starting.
I20260927 00:52:40.580914 2729 tablet_bootstrap.cc:492] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: No bootstrap required, opened a new log
I20260927 00:52:40.580996 2729 ts_tablet_manager.cc:1397] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 00:52:40.581300 2728 tablet_bootstrap.cc:654] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:40.581537 2729 raft_consensus.cc:348] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.581677 2729 raft_consensus.cc:374] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:40.581724 2729 raft_consensus.cc:729] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc92387901e54e7ebde44db98a36f823, State: Initialized, Role: FOLLOWER
I20260927 00:52:40.581842 2729 consensus_queue.cc:260] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.582203 2729 ts_tablet_manager.cc:1428] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:52:40.582410 2727 tablet_bootstrap.cc:492] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: No bootstrap required, opened a new log
I20260927 00:52:40.582520 2727 ts_tablet_manager.cc:1397] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 00:52:40.582937 2728 tablet_bootstrap.cc:492] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: No bootstrap required, opened a new log
I20260927 00:52:40.583037 2728 ts_tablet_manager.cc:1397] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:40.582971 2727 raft_consensus.cc:348] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.583097 2727 raft_consensus.cc:374] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:40.583180 2727 raft_consensus.cc:729] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Initialized, Role: FOLLOWER
I20260927 00:52:40.583365 2727 consensus_queue.cc:260] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.583526 2728 raft_consensus.cc:348] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.583635 2728 raft_consensus.cc:374] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:40.583663 2728 raft_consensus.cc:729] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Initialized, Role: FOLLOWER
I20260927 00:52:40.583789 2727 ts_tablet_manager.cc:1428] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:52:40.583770 2728 consensus_queue.cc:260] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.584146 2728 ts_tablet_manager.cc:1428] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:52:40.763062 2734 raft_consensus.cc:482] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:40.763285 2734 raft_consensus.cc:504] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.764779 2734 leader_election.cc:288] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759), 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:40.769410 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.765281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38591 (local address 127.0.98.3:39943)
0927 00:52:40.765629 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:40.765638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:40.765730 (+ 92us) server_negotiation.cc:407] Connection header received
0927 00:52:40.765888 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:40.765891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:40.765933 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:40.766028 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:40.766349 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.766892 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.768332 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.768735 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.768854 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.768922 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:40.769191 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 00:52:40.769264 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
I20260927 00:52:40.769438 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.765230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42029 (local address 127.0.98.2:43759)
0927 00:52:40.765653 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:40.765662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:40.765682 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:52:40.765833 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:40.765836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:40.765893 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:40.765972 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:40.766323 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.766900 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.768293 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.768730 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.768826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.768923 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:40.769163 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 00:52:40.769252 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":110,"threads_started":1}
I20260927 00:52:40.769452 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.765149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.98.1:38591)
0927 00:52:40.765601 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:40.765618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:40.765766 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:40.766055 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:40.766058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:40.766107 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:40.766234 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.766240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.767005 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.767008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.768187 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.768196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.768893 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.768897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.769059 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.769081 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:52:40.769157 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:52:40.769252 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":131,"spinlock_wait_cycles":7296,"thread_start_us":172,"threads_started":1}
I20260927 00:52:40.769467 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.765121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43759 (local address 127.0.98.1:42029)
0927 00:52:40.765538 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:40.765556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:40.765695 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:40.766016 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:40.766020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:40.766038 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:40.766204 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.766210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.767019 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.767022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.768161 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.768171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.768880 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.768884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.769025 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.769055 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:52:40.769125 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:52:40.769225 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":62,"spinlock_wait_cycles":28928,"thread_start_us":148,"threads_started":1}
I20260927 00:52:40.770026 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13262cee93a5422a8d5b6d92bf1289a5" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639c00e62d344b49bcd8121cb520dcee" is_pre_election: true
I20260927 00:52:40.770187 2566 raft_consensus.cc:2454] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 0.
I20260927 00:52:40.770515 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13262cee93a5422a8d5b6d92bf1289a5" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823" is_pre_election: true
I20260927 00:52:40.770584 2217 leader_election.cc:302] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, ce4c5f0c7e1e40c3bad55067b70e55b7; no voters:
I20260927 00:52:40.770673 2424 raft_consensus.cc:2454] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 0.
I20260927 00:52:40.770866 2734 raft_consensus.cc:2781] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:40.770927 2734 raft_consensus.cc:482] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:40.770967 2734 raft_consensus.cc:3037] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:40.772019 2734 raft_consensus.cc:504] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.772369 2734 leader_election.cc:288] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 election: Requested vote from peers dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759), 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:40.772720 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13262cee93a5422a8d5b6d92bf1289a5" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639c00e62d344b49bcd8121cb520dcee"
I20260927 00:52:40.772809 2566 raft_consensus.cc:3037] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:40.772773 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13262cee93a5422a8d5b6d92bf1289a5" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823"
I20260927 00:52:40.772918 2424 raft_consensus.cc:3037] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:40.773809 2566 raft_consensus.cc:2454] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 1.
I20260927 00:52:40.774094 2217 leader_election.cc:302] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c00e62d344b49bcd8121cb520dcee, ce4c5f0c7e1e40c3bad55067b70e55b7; no voters:
I20260927 00:52:40.774158 2424 raft_consensus.cc:2454] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 1.
I20260927 00:52:40.774259 2734 raft_consensus.cc:2781] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:40.774488 2734 raft_consensus.cc:686] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 LEADER]: Becoming Leader. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Running, Role: LEADER
I20260927 00:52:40.774756 2734 consensus_queue.cc:237] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.777134 2127 catalog_manager.cc:5495] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } health_report { overall_health: UNKNOWN } } }
I20260927 00:52:40.834664 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.830234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.0.98.1:39385)
0927 00:52:40.830672 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:40.830681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:40.830693 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:52:40.830763 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:40.830765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:40.830817 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:40.830917 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:40.831379 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.831954 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.833065 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.833303 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.833386 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.833676 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:40.834195 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:40.834287 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:40.834471 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:52:40.834522 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":166,"threads_started":1}
I20260927 00:52:40.846112 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.842803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39943 (local address 127.0.98.1:36925)
0927 00:52:40.843113 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:40.843128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:40.843260 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:40.843481 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:40.843484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:40.843491 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:40.843617 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.843622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.844358 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.844361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.844984 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.844992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.845416 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.845419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.845496 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.845543 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:40.845816 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:40.845822 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:52:40.845880 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 00:52:40.845956 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260927 00:52:40.846088 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.843082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36925 (local address 127.0.98.3:39943)
0927 00:52:40.843284 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:40.843289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:40.843304 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:52:40.843355 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:40.843357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:40.843402 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:40.843463 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:40.843713 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.844253 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.845093 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.845306 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.845389 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.845673 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:40.845746 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:40.845802 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:40.845888 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 00:52:40.845939 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 00:52:40.854488 2566 raft_consensus.cc:1261] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Refusing update from remote peer ce4c5f0c7e1e40c3bad55067b70e55b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:40.854599 2424 raft_consensus.cc:1261] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Refusing update from remote peer ce4c5f0c7e1e40c3bad55067b70e55b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:40.855000 2734 consensus_queue.cc:1035] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:40.855250 2739 consensus_queue.cc:1035] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:40.876791 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.872484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.98.3:60795)
0927 00:52:40.872753 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:40.872768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:40.872868 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:40.873195 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:40.873198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:40.873209 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:52:40.873320 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.873324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.874218 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.874221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.875107 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:40.875117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.875742 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.875745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:40.875830 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.875861 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:52:40.876466 (+ 605us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:52:40.876475 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:52:40.876552 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:52:40.876647 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260927 00:52:40.876819 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:40.872603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60795 (local address 127.0.98.1:39385)
0927 00:52:40.872848 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:40.872853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:40.872870 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:52:40.872962 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:40.872963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:40.873004 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:40.873073 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:40.873488 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.874071 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.875276 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:40.875568 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:40.875662 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:40.876042 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:40.876160 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:40.876236 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:40.876571 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 00:52:40.876659 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 00:52:40.901165 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:40.901320 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:40.904083 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Sending DeleteTablet for 3 replicas of tablet 13262cee93a5422a8d5b6d92bf1289a5
I20260927 00:52:40.904729 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 13262cee93a5422a8d5b6d92bf1289a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46472
I20260927 00:52:40.904743 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 13262cee93a5422a8d5b6d92bf1289a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260927 00:52:40.904927 2717 tablet_replica.cc:330] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: stopping tablet replica
I20260927 00:52:40.904860 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 13262cee93a5422a8d5b6d92bf1289a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33588
I20260927 00:52:40.905035 2717 raft_consensus.cc:2229] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:40.905151 2717 raft_consensus.cc:2258] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:40.905138 2716 tablet_replica.cc:330] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: stopping tablet replica
I20260927 00:52:40.905267 2716 raft_consensus.cc:2229] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:40.905309 2718 tablet_replica.cc:330] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: stopping tablet replica
I20260927 00:52:40.905421 2718 raft_consensus.cc:2229] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:40.905550 2718 raft_consensus.cc:2258] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:40.905592 2716 raft_consensus.cc:2258] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:40.906378 2718 ts_tablet_manager.cc:1905] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:40.906371 2716 ts_tablet_manager.cc:1905] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:40.906538 2717 ts_tablet_manager.cc:1905] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:40.908981 2716 ts_tablet_manager.cc:1918] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:40.908981 2718 ts_tablet_manager.cc:1918] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:40.909066 2718 log.cc:1196] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/13262cee93a5422a8d5b6d92bf1289a5
I20260927 00:52:40.909066 2716 log.cc:1196] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/13262cee93a5422a8d5b6d92bf1289a5
I20260927 00:52:40.909312 2716 ts_tablet_manager.cc:1939] T 13262cee93a5422a8d5b6d92bf1289a5 P ce4c5f0c7e1e40c3bad55067b70e55b7: Deleting consensus metadata
I20260927 00:52:40.909312 2718 ts_tablet_manager.cc:1939] T 13262cee93a5422a8d5b6d92bf1289a5 P 639c00e62d344b49bcd8121cb520dcee: Deleting consensus metadata
I20260927 00:52:40.910053 2113 catalog_manager.cc:4841] TS 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943): tablet 13262cee93a5422a8d5b6d92bf1289a5 (table test-table [id=0c7855fb79c44e4c805d32b33fc9e968]) successfully deleted
I20260927 00:52:40.910328 2116 catalog_manager.cc:4841] TS ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385): tablet 13262cee93a5422a8d5b6d92bf1289a5 (table test-table [id=0c7855fb79c44e4c805d32b33fc9e968]) successfully deleted
I20260927 00:52:40.910858 2717 ts_tablet_manager.cc:1918] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:52:40.910969 2717 log.cc:1196] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/13262cee93a5422a8d5b6d92bf1289a5
I20260927 00:52:40.911276 2717 ts_tablet_manager.cc:1939] T 13262cee93a5422a8d5b6d92bf1289a5 P dc92387901e54e7ebde44db98a36f823: Deleting consensus metadata
I20260927 00:52:40.911206 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:40.911648 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:40.912402 2115 catalog_manager.cc:4841] TS dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759): tablet 13262cee93a5422a8d5b6d92bf1289a5 (table test-table [id=0c7855fb79c44e4c805d32b33fc9e968]) successfully deleted
I20260927 00:52:40.917001 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 5b422c8e741b4a0ab9ef38a65b7ac025 (DEFAULT_TABLE table=test-table [id=a3117fd7384a417684aebc5b18c5b959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:40.917311 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b422c8e741b4a0ab9ef38a65b7ac025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:40.917279 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 5b422c8e741b4a0ab9ef38a65b7ac025 (DEFAULT_TABLE table=test-table [id=a3117fd7384a417684aebc5b18c5b959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:40.917637 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b422c8e741b4a0ab9ef38a65b7ac025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:40.919569 2729 tablet_bootstrap.cc:492] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: Bootstrap starting.
I20260927 00:52:40.920369 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 5b422c8e741b4a0ab9ef38a65b7ac025 (DEFAULT_TABLE table=test-table [id=a3117fd7384a417684aebc5b18c5b959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:40.920455 2729 tablet_bootstrap.cc:654] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:40.920631 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b422c8e741b4a0ab9ef38a65b7ac025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:40.920673 2727 tablet_bootstrap.cc:492] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: Bootstrap starting.
I20260927 00:52:40.921496 2729 tablet_bootstrap.cc:492] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: No bootstrap required, opened a new log
I20260927 00:52:40.921579 2729 ts_tablet_manager.cc:1397] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 00:52:40.921689 2727 tablet_bootstrap.cc:654] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:40.922005 2729 raft_consensus.cc:348] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.922116 2729 raft_consensus.cc:374] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:40.922149 2729 raft_consensus.cc:729] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc92387901e54e7ebde44db98a36f823, State: Initialized, Role: FOLLOWER
I20260927 00:52:40.922267 2729 consensus_queue.cc:260] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.922511 2729 ts_tablet_manager.cc:1428] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:52:40.923363 2728 tablet_bootstrap.cc:492] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7: Bootstrap starting.
I20260927 00:52:40.924330 2728 tablet_bootstrap.cc:654] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:40.924867 2727 tablet_bootstrap.cc:492] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: No bootstrap required, opened a new log
I20260927 00:52:40.924978 2727 ts_tablet_manager.cc:1397] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 00:52:40.925431 2727 raft_consensus.cc:348] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.925567 2727 raft_consensus.cc:374] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:40.925609 2727 raft_consensus.cc:729] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c00e62d344b49bcd8121cb520dcee, State: Initialized, Role: FOLLOWER
I20260927 00:52:40.925752 2727 consensus_queue.cc:260] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.925858 2728 tablet_bootstrap.cc:492] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7: No bootstrap required, opened a new log
I20260927 00:52:40.925973 2728 ts_tablet_manager.cc:1397] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:52:40.926437 2728 raft_consensus.cc:348] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.926545 2728 raft_consensus.cc:374] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:40.926573 2727 ts_tablet_manager.cc:1428] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 00:52:40.926576 2728 raft_consensus.cc:729] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Initialized, Role: FOLLOWER
I20260927 00:52:40.926703 2728 consensus_queue.cc:260] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:40.926935 2728 ts_tablet_manager.cc:1428] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:52:41.133117 2739 raft_consensus.cc:482] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:41.133298 2739 raft_consensus.cc:504] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:41.133755 2739 leader_election.cc:288] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759), 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:41.134202 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b422c8e741b4a0ab9ef38a65b7ac025" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823" is_pre_election: true
I20260927 00:52:41.134348 2424 raft_consensus.cc:2454] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 0.
I20260927 00:52:41.134361 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b422c8e741b4a0ab9ef38a65b7ac025" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639c00e62d344b49bcd8121cb520dcee" is_pre_election: true
I20260927 00:52:41.134586 2566 raft_consensus.cc:2454] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 0.
I20260927 00:52:41.134671 2219 leader_election.cc:302] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce4c5f0c7e1e40c3bad55067b70e55b7, dc92387901e54e7ebde44db98a36f823; no voters:
I20260927 00:52:41.134840 2739 raft_consensus.cc:2781] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:41.134892 2739 raft_consensus.cc:482] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:41.134928 2739 raft_consensus.cc:3037] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:41.136021 2739 raft_consensus.cc:504] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:41.136346 2739 leader_election.cc:288] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 election: Requested vote from peers dc92387901e54e7ebde44db98a36f823 (127.0.98.2:43759), 639c00e62d344b49bcd8121cb520dcee (127.0.98.3:39943)
I20260927 00:52:41.136642 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b422c8e741b4a0ab9ef38a65b7ac025" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92387901e54e7ebde44db98a36f823"
I20260927 00:52:41.136747 2424 raft_consensus.cc:3037] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:41.136829 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b422c8e741b4a0ab9ef38a65b7ac025" candidate_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "639c00e62d344b49bcd8121cb520dcee"
I20260927 00:52:41.137012 2566 raft_consensus.cc:3037] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:41.137681 2424 raft_consensus.cc:2454] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 1.
I20260927 00:52:41.137957 2219 leader_election.cc:302] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce4c5f0c7e1e40c3bad55067b70e55b7, dc92387901e54e7ebde44db98a36f823; no voters:
I20260927 00:52:41.138111 2739 raft_consensus.cc:2781] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:41.138185 2739 raft_consensus.cc:686] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [term 1 LEADER]: Becoming Leader. State: Replica: ce4c5f0c7e1e40c3bad55067b70e55b7, State: Running, Role: LEADER
I20260927 00:52:41.138316 2739 consensus_queue.cc:237] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } }
I20260927 00:52:41.138446 2566 raft_consensus.cc:2454] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce4c5f0c7e1e40c3bad55067b70e55b7 in term 1.
I20260927 00:52:41.140177 2128 catalog_manager.cc:5495] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce4c5f0c7e1e40c3bad55067b70e55b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 } health_report { overall_health: UNKNOWN } } }
I20260927 00:52:41.176970 2566 raft_consensus.cc:1261] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Refusing update from remote peer ce4c5f0c7e1e40c3bad55067b70e55b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:41.176941 2424 raft_consensus.cc:1261] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Refusing update from remote peer ce4c5f0c7e1e40c3bad55067b70e55b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:41.177446 2734 consensus_queue.cc:1035] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc92387901e54e7ebde44db98a36f823" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:41.177657 2739 consensus_queue.cc:1035] T 5b422c8e741b4a0ab9ef38a65b7ac025 P ce4c5f0c7e1e40c3bad55067b70e55b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "639c00e62d344b49bcd8121cb520dcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:52:41.193171 2701 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:52:42.750204 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:42.750460 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:42.753513 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1238144f3e3426e80b128f198d1e04a: Sending DeleteTablet for 3 replicas of tablet 5b422c8e741b4a0ab9ef38a65b7ac025
I20260927 00:52:42.754261 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b422c8e741b4a0ab9ef38a65b7ac025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33588
I20260927 00:52:42.754261 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b422c8e741b4a0ab9ef38a65b7ac025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46472
I20260927 00:52:42.754644 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b422c8e741b4a0ab9ef38a65b7ac025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260927 00:52:42.754989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 2201
I20260927 00:52:42.755832 2781 tablet_replica.cc:330] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: stopping tablet replica
I20260927 00:52:42.755978 2781 raft_consensus.cc:2229] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:42.756110 2781 raft_consensus.cc:2258] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:42.756840 2781 ts_tablet_manager.cc:1905] T 5b422c8e741b4a0ab9ef38a65b7ac025 P 639c00e62d344b49bcd8121cb520dcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:42.758313 2779 tablet_replica.cc:330] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: stopping tablet replica
I20260927 00:52:42.758499 2779 raft_consensus.cc:2229] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:42.758651 2779 raft_consensus.cc:2258] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:42.759564 2779 ts_tablet_manager.cc:1905] T 5b422c8e741b4a0ab9ef38a65b7ac025 P dc92387901e54e7ebde44db98a36f823: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:52:42.765890 2116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39385 (error 108)
I20260927 00:52:42.765973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 2341
I20260927 00:52:42.768467 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:42.767801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.0.1:52500)
0927 00:52:42.768236 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:42.768349 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":184,"threads_started":1}
W20260927 00:52:42.768667 2116 catalog_manager.cc:4584] TS ce4c5f0c7e1e40c3bad55067b70e55b7 (127.0.98.1:39385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b422c8e741b4a0ab9ef38a65b7ac025: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39385: connect: Connection refused (error 111)
I20260927 00:52:42.774880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 2481
I20260927 00:52:42.776499 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:42.774708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43759 (local address 127.0.0.1:59514)
0927 00:52:42.774874 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:42.774933 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:52:42.782045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 2097
2026-09-27T00:52:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7083 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:44 dist-test-slave-9fh2 krb5kdc[2786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:44 dist-test-slave-9fh2 krb5kdc[2786](info): set up 2 sockets
Sep 27 00:52:44 dist-test-slave-9fh2 krb5kdc[2786](info): commencing operation
krb5kdc: starting...
W20260927 00:52:44.959339 392 mini_kdc.cc:121] Time spent starting KDC: real 2.151s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:45.095654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:45.218869 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:45.219187 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:45.219231 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:45.223331 2799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:45.223410 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:45.223430 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:45.223445 2799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:45.223464 2799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:45.229010 2799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:45.230190 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:45.231302 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:52:45.241277 2799 server_base.cc:1034] running on GCE node
W20260927 00:52:45.241219 2805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:45.241232 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:45.241232 2807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:45.242274 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:45.243501 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:45.244694 2799 hybrid_clock.cc:648] HybridClock initialized: now 1790470365244663 us; error 60 us; skew 500 ppm
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:45.247848 2799 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:45.248899 2799 webserver.cc:460] Webserver started at http://127.0.98.62:45381/ using document root <none> and password file <none>
I20260927 00:52:45.249558 2799 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:45.249644 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:45.249863 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:45.252094 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "40c87fd1c7364116a006b8eb507253ea"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "e97bf50ae9bcbd1401c01cc588bc1915"
server_key_iv: "a42b4b8650d61bf97515fdac2dde81ea"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.252678 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "40c87fd1c7364116a006b8eb507253ea"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "e97bf50ae9bcbd1401c01cc588bc1915"
server_key_iv: "a42b4b8650d61bf97515fdac2dde81ea"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.257158 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:45.260107 2814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.261718 2799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:52:45.261865 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "40c87fd1c7364116a006b8eb507253ea"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "e97bf50ae9bcbd1401c01cc588bc1915"
server_key_iv: "a42b4b8650d61bf97515fdac2dde81ea"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.262001 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:45.280982 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:45.284412 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:45.284653 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:45.292186 2799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42195
I20260927 00:52:45.292306 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42195 every 8 connection(s)
I20260927 00:52:45.293226 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:45.294145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 2799
I20260927 00:52:45.294281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:45.294621 392 external_mini_cluster.cc:1428] Setting key c351df20c396973e2bea36efa296333f
I20260927 00:52:45.297411 2867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:45.304472 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea: Bootstrap starting.
I20260927 00:52:45.307273 2867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:45.308105 2867 log.cc:824] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:45.309289 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:45.296451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.0.98.62:42195)
0927 00:52:45.297020 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:45.297036 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:45.297073 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:52:45.297824 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:45.297844 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:45.298232 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:45.298564 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:45.299388 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.300431 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.301257 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.301746 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.301889 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:45.304712 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:45.304718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:45.304732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:45.304761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:45.306368 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:45.306760 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:45.306765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:45.306770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:45.306917 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:45.307174 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:45.307177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:45.307179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:45.307606 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:45.307794 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:45.308067 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 00:52:45.308370 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":204,"threads_started":1}
I20260927 00:52:45.310928 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea: No bootstrap required, opened a new log
I20260927 00:52:45.315037 2867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c87fd1c7364116a006b8eb507253ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 00:52:45.315382 2867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:45.315445 2867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c87fd1c7364116a006b8eb507253ea, State: Initialized, Role: FOLLOWER
I20260927 00:52:45.315937 2867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c87fd1c7364116a006b8eb507253ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 00:52:45.316089 2867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:45.316159 2867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:45.316272 2867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:45.317530 2867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c87fd1c7364116a006b8eb507253ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 00:52:45.317922 2867 leader_election.cc:302] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c87fd1c7364116a006b8eb507253ea; no voters:
I20260927 00:52:45.318270 2867 leader_election.cc:288] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:45.318444 2872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:45.318719 2872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [term 1 LEADER]: Becoming Leader. State: Replica: 40c87fd1c7364116a006b8eb507253ea, State: Running, Role: LEADER
I20260927 00:52:45.319178 2872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c87fd1c7364116a006b8eb507253ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 00:52:45.319697 2867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:45.320960 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c87fd1c7364116a006b8eb507253ea. Latest consensus state: current_term: 1 leader_uuid: "40c87fd1c7364116a006b8eb507253ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c87fd1c7364116a006b8eb507253ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } } }
I20260927 00:52:45.321107 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:45.321503 2873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c87fd1c7364116a006b8eb507253ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c87fd1c7364116a006b8eb507253ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } } }
I20260927 00:52:45.321552 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:45.321709 2873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:45.324977 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:45.331614 2881 catalog_manager.cc:1347] Generated new cluster ID: 45409de633ef4d6395e9915661714471
I20260927 00:52:45.331715 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:45.347069 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:45.348300 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:45.359108 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c87fd1c7364116a006b8eb507253ea: Generated new TSK 0
I20260927 00:52:45.360363 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:45.395363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42195
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:45.505657 2893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:45.505926 2893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:45.505970 2893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:45.510093 2893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:45.510169 2893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:45.510227 2893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:45.515532 2893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:45.516561 2893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:45.517560 2893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:52:45.527829 2893 server_base.cc:1034] running on GCE node
W20260927 00:52:45.527787 2898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:45.527787 2899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:45.527784 2901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:45.528671 2893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:45.529425 2893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:45.530598 2893 hybrid_clock.cc:648] HybridClock initialized: now 1790470365530579 us; error 34 us; skew 500 ppm
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:45.533808 2893 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:45.534852 2893 webserver.cc:460] Webserver started at http://127.0.98.1:40281/ using document root <none> and password file <none>
I20260927 00:52:45.535607 2893 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:45.535681 2893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:45.535905 2893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:45.538378 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c50e79ee6c940c09a545dfd2d54ba0f"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "f2b95e285c9b82e3e60dc828538afab8"
server_key_iv: "05cf7c046a05d9bb2cf7047f26061e82"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.539011 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c50e79ee6c940c09a545dfd2d54ba0f"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "f2b95e285c9b82e3e60dc828538afab8"
server_key_iv: "05cf7c046a05d9bb2cf7047f26061e82"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.543623 2893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 00:52:45.547392 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.549361 2893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:52:45.549515 2893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "3c50e79ee6c940c09a545dfd2d54ba0f"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "f2b95e285c9b82e3e60dc828538afab8"
server_key_iv: "05cf7c046a05d9bb2cf7047f26061e82"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.549665 2893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:45.577984 2893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:45.581398 2893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:45.581580 2893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:45.582176 2893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:45.583053 2893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:45.583103 2893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.583184 2893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:45.583214 2893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.592454 2893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43819
I20260927 00:52:45.592505 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43819 every 8 connection(s)
I20260927 00:52:45.593349 2893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:45.603502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 2893
I20260927 00:52:45.603644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:45.604012 392 external_mini_cluster.cc:1428] Setting key d893740276b1a8c9cc27e20279a0d092
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:45.609778 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:45.595689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56283 (local address 127.0.98.62:42195)
0927 00:52:45.595879 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:45.595884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:45.596867 (+ 983us) server_negotiation.cc:407] Connection header received
0927 00:52:45.598432 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:45.598435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:45.598500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:45.598581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:45.600057 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.600650 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.601892 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.602165 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.602257 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:45.605554 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:45.605558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:45.605561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:45.605592 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:45.607433 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:45.608186 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:45.608190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:45.608192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:45.608291 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:45.608620 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:45.608623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:45.608625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:45.608819 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:45.608908 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:45.609448 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 00:52:45.609608 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:52:45.610682 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:45.596067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42195 (local address 127.0.98.1:56283)
0927 00:52:45.596651 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:45.596705 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:45.598183 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:45.598792 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:45.598800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:45.599288 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:45.599900 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:45.599917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.600774 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.600776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:45.601770 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:45.601781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.602345 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.602350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:45.602527 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:45.603245 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:45.603268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:45.605325 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:45.607576 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:45.607584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:45.607601 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:45.608030 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:45.608383 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:45.608387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:45.608389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:45.608533 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:45.608920 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:45.608927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:45.609210 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 00:52:45.609390 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:52:45.609768 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":125,"threads_started":1}
I20260927 00:52:45.612056 3022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42195
I20260927 00:52:45.612488 3022 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:45.613330 3022 heartbeater.cc:511] Master 127.0.98.62:42195 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:45.615252 2829 ts_manager.cc:194] Registered new tserver with Master: 3c50e79ee6c940c09a545dfd2d54ba0f (127.0.98.1:43819)
I20260927 00:52:45.616496 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:45.635743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42195
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:45.762004 3027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:45.762351 3027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:45.762424 3027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:45.766913 3027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:45.767005 3027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:45.767091 3027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:45.773309 3027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:45.774708 3027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:45.775990 3027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:45.787117 3035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:45.787328 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:45.787441 3027 server_base.cc:1034] running on GCE node
W20260927 00:52:45.787120 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:45.787955 3027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:45.788721 3027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:45.789901 3027 hybrid_clock.cc:648] HybridClock initialized: now 1790470365789854 us; error 65 us; skew 500 ppm
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:45.792943 3027 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:45.793972 3027 webserver.cc:460] Webserver started at http://127.0.98.2:39233/ using document root <none> and password file <none>
I20260927 00:52:45.794601 3027 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:45.794668 3027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:45.794867 3027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:45.797129 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "40b651c429b848b08f1f716b3bc268f8"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "bdf6f75307d28005d55add75cddf30eb"
server_key_iv: "214374c5871d951e2d47e4114e57f8a6"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.797691 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "40b651c429b848b08f1f716b3bc268f8"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "bdf6f75307d28005d55add75cddf30eb"
server_key_iv: "214374c5871d951e2d47e4114e57f8a6"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.801927 3027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:52:45.804867 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.806366 3027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:52:45.806496 3027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "40b651c429b848b08f1f716b3bc268f8"
format_stamp: "Formatted at 2026-09-27 00:52:45 on dist-test-slave-9fh2"
server_key: "bdf6f75307d28005d55add75cddf30eb"
server_key_iv: "214374c5871d951e2d47e4114e57f8a6"
server_key_version: "encryptionkey@0"
I20260927 00:52:45.806607 3027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:45.829452 3027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:45.832580 3027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:45.832741 3027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:45.833307 3027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:45.834170 3027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:45.834223 3027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.834264 3027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:45.834278 3027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:45.843732 3027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36787
I20260927 00:52:45.843812 3155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36787 every 8 connection(s)
I20260927 00:52:45.844726 3027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:45.853538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3027
I20260927 00:52:45.853652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:45.853948 392 external_mini_cluster.cc:1428] Setting key 97dcdd792df8aa2fff70f75fe7f51ac1
Sep 27 00:52:45 dist-test-slave-9fh2 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:45.861443 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:45.847002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34605 (local address 127.0.98.62:42195)
0927 00:52:45.847199 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:45.847204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:45.848150 (+ 946us) server_negotiation.cc:407] Connection header received
0927 00:52:45.849867 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:45.849871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:45.849940 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:45.850023 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:45.851532 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.852119 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.853483 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.853804 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.853954 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:45.857499 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:45.857502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:45.857504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:45.857535 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:45.859290 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:45.859877 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:45.859880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:45.859882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:45.859957 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:45.860281 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:45.860284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:45.860286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:45.860486 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:45.860571 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:45.861140 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 00:52:45.861274 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:52:45.862339 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:45.847420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42195 (local address 127.0.98.2:34605)
0927 00:52:45.847958 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:45.848011 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:45.849595 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:45.850242 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:45.850250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:45.850722 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:45.851375 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:45.851393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.852245 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.852248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:45.853287 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:45.853298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:45.854037 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:45.854042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:45.854406 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:45.855313 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:45.855341 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:45.857292 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:45.859437 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:45.859446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:45.859462 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:45.859767 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:45.860036 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:45.860039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:45.860041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:45.860184 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:45.860591 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:45.860598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:45.860885 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 00:52:45.861060 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 00:52:45.861464 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260927 00:52:45.863580 3156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42195
I20260927 00:52:45.863826 3156 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:45.864341 3156 heartbeater.cc:511] Master 127.0.98.62:42195 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:45.865424 2829 ts_manager.cc:194] Registered new tserver with Master: 40b651c429b848b08f1f716b3bc268f8 (127.0.98.2:36787)
I20260927 00:52:45.866041 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:45.886770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42195
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:46.018136 3161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:46.018522 3161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:46.018574 3161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:46.023459 3161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:46.023571 3161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:46.023649 3161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:46.030077 3161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:46.031635 3161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:46.032968 3161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:46.044761 3167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:46.044803 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:46.044761 3166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:46.045176 3161 server_base.cc:1034] running on GCE node
I20260927 00:52:46.045825 3161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:46.046685 3161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:46.047852 3161 hybrid_clock.cc:648] HybridClock initialized: now 1790470366047816 us; error 52 us; skew 500 ppm
Sep 27 00:52:46 dist-test-slave-9fh2 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:46.050973 3161 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:46.052147 3161 webserver.cc:460] Webserver started at http://127.0.98.3:41545/ using document root <none> and password file <none>
I20260927 00:52:46.052875 3161 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:46.052948 3161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:46.053216 3161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:46.055791 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "cbce2d32fc004a7faf782c63f77ad9f2"
format_stamp: "Formatted at 2026-09-27 00:52:46 on dist-test-slave-9fh2"
server_key: "ff62ab35a0e43b141c52c91d0515fe61"
server_key_iv: "408704869e9078690868e0a22bc71e15"
server_key_version: "encryptionkey@0"
I20260927 00:52:46.056406 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cbce2d32fc004a7faf782c63f77ad9f2"
format_stamp: "Formatted at 2026-09-27 00:52:46 on dist-test-slave-9fh2"
server_key: "ff62ab35a0e43b141c52c91d0515fe61"
server_key_iv: "408704869e9078690868e0a22bc71e15"
server_key_version: "encryptionkey@0"
I20260927 00:52:46.061039 3161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:46.064188 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:46.066025 3161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:52:46.066172 3161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "cbce2d32fc004a7faf782c63f77ad9f2"
format_stamp: "Formatted at 2026-09-27 00:52:46 on dist-test-slave-9fh2"
server_key: "ff62ab35a0e43b141c52c91d0515fe61"
server_key_iv: "408704869e9078690868e0a22bc71e15"
server_key_version: "encryptionkey@0"
I20260927 00:52:46.066308 3161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:46.082204 3161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:46.085979 3161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:46.086198 3161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:46.086918 3161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:46.087939 3161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:46.087991 3161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:46.088050 3161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:46.088078 3161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:46.098850 3161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43417
I20260927 00:52:46.098898 3289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43417 every 8 connection(s)
I20260927 00:52:46.099979 3161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:46.105351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3161
I20260927 00:52:46.105489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:46.105799 392 external_mini_cluster.cc:1428] Setting key d548811f8ace113e3678e3372f3fd44b
Sep 27 00:52:46 dist-test-slave-9fh2 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:46.114377 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:46.102200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45413 (local address 127.0.98.62:42195)
0927 00:52:46.102330 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:46.102334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:46.103416 (+ 1082us) server_negotiation.cc:407] Connection header received
0927 00:52:46.104925 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:46.104929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:46.104983 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:46.105092 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:46.106488 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:46.107098 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:46.108291 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:46.108500 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:46.108584 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:46.111070 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:46.111073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:46.111076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:46.111102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:46.112446 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:46.112926 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:46.112929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:46.112931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:46.113008 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:46.113314 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:46.113318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:46.113319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:46.113497 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:46.113580 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:46.114089 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 00:52:46.114210 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:52:46.115270 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:46.102641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42195 (local address 127.0.98.3:45413)
0927 00:52:46.103249 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:46.103301 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:46.104708 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:46.105280 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:46.105287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:46.105731 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:46.106326 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:46.106352 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:46.107246 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:46.107250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:46.108173 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:46.108183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:46.108641 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:46.108645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:46.108804 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:46.109384 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:46.109400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:46.110880 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:46.112567 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:46.112573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:46.112590 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:46.112832 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:46.113095 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:46.113098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:46.113100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:46.113230 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:46.113588 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:46.113595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:46.113849 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 00:52:46.114024 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 00:52:46.114354 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":130,"threads_started":1}
I20260927 00:52:46.116492 3290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42195
I20260927 00:52:46.116731 3290 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:46.117249 3290 heartbeater.cc:511] Master 127.0.98.62:42195 requested a full tablet report, sending...
I20260927 00:52:46.118391 2829 ts_manager.cc:194] Registered new tserver with Master: cbce2d32fc004a7faf782c63f77ad9f2 (127.0.98.3:43417)
I20260927 00:52:46.119284 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45413
I20260927 00:52:46.120141 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:52:46.134618 2870 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:46.133066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.0.98.62:42195)
0927 00:52:46.133223 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:46.133227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:46.133294 (+ 67us) server_negotiation.cc:407] Connection header received
0927 00:52:46.133478 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:46.133481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:46.133538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:46.133639 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:46.134463 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54186: BlockingRecv error: recv got EOF from 127.0.0.1:54186 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:52:46.135653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 2893
I20260927 00:52:46.141031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3027
I20260927 00:52:46.145908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3161
I20260927 00:52:46.150827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 2799
2026-09-27T00:52:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3364 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:46 dist-test-slave-9fh2 krb5kdc[3302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:46 dist-test-slave-9fh2 krb5kdc[3302](info): set up 2 sockets
Sep 27 00:52:46 dist-test-slave-9fh2 krb5kdc[3302](info): commencing operation
krb5kdc: starting...
W20260927 00:52:47.123974 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:47 dist-test-slave-9fh2 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:47.263012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:47.383096 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:47.383406 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:47.383447 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:47.387722 3315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:47.387801 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:47.387820 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:47.387837 3315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:47.387851 3315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:47.393419 3315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:47.394552 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:47.395632 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:47.404451 3323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:47.404436 3321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:47.404511 3315 server_base.cc:1034] running on GCE node
W20260927 00:52:47.404426 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:47.405377 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:47.406531 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:47.407707 3315 hybrid_clock.cc:648] HybridClock initialized: now 1790470367407682 us; error 55 us; skew 500 ppm
Sep 27 00:52:47 dist-test-slave-9fh2 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:47.410650 3315 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:47.411662 3315 webserver.cc:460] Webserver started at http://127.0.98.62:37035/ using document root <none> and password file <none>
I20260927 00:52:47.412281 3315 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:47.412348 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:47.412539 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:47.414606 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "a55e9c20eda14422ac5f1d7fdf2043a5"
format_stamp: "Formatted at 2026-09-27 00:52:47 on dist-test-slave-9fh2"
server_key: "019e170282ef9f1a564f1b72549234a5"
server_key_iv: "dcba9e58c6222c9f26c80e1317757ca5"
server_key_version: "encryptionkey@0"
I20260927 00:52:47.415172 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "a55e9c20eda14422ac5f1d7fdf2043a5"
format_stamp: "Formatted at 2026-09-27 00:52:47 on dist-test-slave-9fh2"
server_key: "019e170282ef9f1a564f1b72549234a5"
server_key_iv: "dcba9e58c6222c9f26c80e1317757ca5"
server_key_version: "encryptionkey@0"
I20260927 00:52:47.419375 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:47.422381 3330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:47.424144 3315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:47.424297 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "a55e9c20eda14422ac5f1d7fdf2043a5"
format_stamp: "Formatted at 2026-09-27 00:52:47 on dist-test-slave-9fh2"
server_key: "019e170282ef9f1a564f1b72549234a5"
server_key_iv: "dcba9e58c6222c9f26c80e1317757ca5"
server_key_version: "encryptionkey@0"
I20260927 00:52:47.424408 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:47.444505 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:47.447600 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:47.447777 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:47.454967 3315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35921
I20260927 00:52:47.455345 3382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35921 every 8 connection(s)
I20260927 00:52:47.456039 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:47.459595 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:47.460466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3315
I20260927 00:52:47.460590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:47.460860 392 external_mini_cluster.cc:1428] Setting key 2bb43d28a8c5b5307c6531587eb81e8f
I20260927 00:52:47.467483 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5: Bootstrap starting.
I20260927 00:52:47.470546 3383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5: Neither blocks nor log segments found. Creating new log.
Sep 27 00:52:47 dist-test-slave-9fh2 krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:47.471534 3383 log.cc:824] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:47.474284 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5: No bootstrap required, opened a new log
I20260927 00:52:47.477324 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:47.462398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.98.62:35921)
0927 00:52:47.462899 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:47.462921 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:47.462974 (+ 53us) server_negotiation.cc:407] Connection header received
0927 00:52:47.463912 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:47.463939 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:47.464437 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:47.464858 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:47.465625 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:47.467187 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:47.468116 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:47.468837 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:47.469018 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:47.471693 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:47.471700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:47.471718 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:47.471753 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:47.473805 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:47.474287 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:47.474294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:47.474300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:47.474457 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:47.474757 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:47.474760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:47.474762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:47.475355 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:47.475568 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:47.475815 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 00:52:47.476151 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260927 00:52:47.479038 3383 raft_consensus.cc:348] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 00:52:47.479343 3383 raft_consensus.cc:374] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:47.479403 3383 raft_consensus.cc:729] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55e9c20eda14422ac5f1d7fdf2043a5, State: Initialized, Role: FOLLOWER
I20260927 00:52:47.479924 3383 consensus_queue.cc:260] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 00:52:47.480073 3383 raft_consensus.cc:388] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:47.480140 3383 raft_consensus.cc:482] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:47.480242 3383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:47.480569 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:52:47.481602 3383 raft_consensus.cc:504] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 00:52:47.482041 3383 leader_election.cc:302] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a55e9c20eda14422ac5f1d7fdf2043a5; no voters:
I20260927 00:52:47.482431 3383 leader_election.cc:288] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:47.482582 3388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:47.482865 3388 raft_consensus.cc:686] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [term 1 LEADER]: Becoming Leader. State: Replica: a55e9c20eda14422ac5f1d7fdf2043a5, State: Running, Role: LEADER
I20260927 00:52:47.483278 3388 consensus_queue.cc:237] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260927 00:52:47.483990 3383 sys_catalog.cc:564] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:47.485126 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260927 00:52:47.485284 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:47.485603 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a55e9c20eda14422ac5f1d7fdf2043a5. Latest consensus state: current_term: 1 leader_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55e9c20eda14422ac5f1d7fdf2043a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260927 00:52:47.485682 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5 [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:47.489568 3404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:47.489667 3404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:47.489835 3398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:47.490819 3398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:47.498013 3398 catalog_manager.cc:1347] Generated new cluster ID: 4175fb16bef04637be7a72b886792b53
I20260927 00:52:47.498090 3398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:47.505825 3398 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:47.507280 3398 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:47.513684 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:47.507952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.98.62:35921)
0927 00:52:47.508081 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:47.508086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:47.508106 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:52:47.508162 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:47.508166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:47.508213 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:47.508354 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:47.508968 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:47.509800 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:47.511037 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:47.511435 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:47.511532 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:47.512510 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:47.512514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:47.512517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:47.513186 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:47.513225 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:47.513244 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:47.513332 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:47.513445 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 00:52:47.513515 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:52:47.519203 3398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a55e9c20eda14422ac5f1d7fdf2043a5: Generated new TSK 0
I20260927 00:52:47.520222 3398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:52:47.532018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3315
2026-09-27T00:52:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1380 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:47 dist-test-slave-9fh2 krb5kdc[3416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:47 dist-test-slave-9fh2 krb5kdc[3416](info): set up 2 sockets
Sep 27 00:52:47 dist-test-slave-9fh2 krb5kdc[3416](info): commencing operation
krb5kdc: starting...
W20260927 00:52:48.495456 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:48 dist-test-slave-9fh2 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:48.627589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:48.741255 3429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:48.741606 3429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:48.741669 3429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:48.746270 3429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:48.746371 3429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:48.746400 3429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:48.746421 3429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:48.746443 3429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:48.752821 3429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:48.754207 3429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:48.755486 3429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:48.765450 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:48.765394 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:48.765467 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:48.765715 3429 server_base.cc:1034] running on GCE node
I20260927 00:52:48.766438 3429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:48.767937 3429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:48.769151 3429 hybrid_clock.cc:648] HybridClock initialized: now 1790470368769108 us; error 65 us; skew 500 ppm
Sep 27 00:52:48 dist-test-slave-9fh2 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:48.772702 3429 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:48.773916 3429 webserver.cc:460] Webserver started at http://127.0.98.62:40631/ using document root <none> and password file <none>
I20260927 00:52:48.774598 3429 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:48.774663 3429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:48.774870 3429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:48.777415 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "81dee74716694b1eaae8542d9324d34f"
format_stamp: "Formatted at 2026-09-27 00:52:48 on dist-test-slave-9fh2"
server_key: "1909d7113fcc20c1bf19a2b0237b2f6f"
server_key_iv: "b633a4418bea97e738822aa2d6ffdb84"
server_key_version: "encryptionkey@0"
I20260927 00:52:48.777997 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "81dee74716694b1eaae8542d9324d34f"
format_stamp: "Formatted at 2026-09-27 00:52:48 on dist-test-slave-9fh2"
server_key: "1909d7113fcc20c1bf19a2b0237b2f6f"
server_key_iv: "b633a4418bea97e738822aa2d6ffdb84"
server_key_version: "encryptionkey@0"
I20260927 00:52:48.782529 3429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:52:48.785606 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:48.787320 3429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:48.787467 3429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "81dee74716694b1eaae8542d9324d34f"
format_stamp: "Formatted at 2026-09-27 00:52:48 on dist-test-slave-9fh2"
server_key: "1909d7113fcc20c1bf19a2b0237b2f6f"
server_key_iv: "b633a4418bea97e738822aa2d6ffdb84"
server_key_version: "encryptionkey@0"
I20260927 00:52:48.787596 3429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:48.809625 3429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:48.813187 3429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:48.813385 3429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:48.821632 3429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46359
I20260927 00:52:48.821942 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46359 every 8 connection(s)
I20260927 00:52:48.822865 3429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:48.825218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3429
I20260927 00:52:48.825357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:48.825656 392 external_mini_cluster.cc:1428] Setting key 3323fd3b15e60aeb9533889a09510545
I20260927 00:52:48.826402 3497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:48.833884 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f: Bootstrap starting.
Sep 27 00:52:48 dist-test-slave-9fh2 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:48.836592 3497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:48.837319 3497 log.cc:824] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:48.839277 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f: No bootstrap required, opened a new log
I20260927 00:52:48.842413 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:48.827489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.0.98.62:46359)
0927 00:52:48.828042 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:48.828061 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:48.828109 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:52:48.828929 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:48.828950 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:48.829393 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:48.829767 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:48.830701 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:48.832051 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:48.833253 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:48.833872 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:48.834025 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:48.837041 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:48.837047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:48.837067 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:48.837104 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:48.838898 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:48.839478 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:48.839487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:48.839494 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:48.839686 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:48.839992 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:48.839995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:48.839997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:48.840533 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:48.840755 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:48.841032 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:52:48.841380 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":178,"threads_started":1}
I20260927 00:52:48.843600 3497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dee74716694b1eaae8542d9324d34f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46359 } }
I20260927 00:52:48.843833 3497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:48.843895 3497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dee74716694b1eaae8542d9324d34f, State: Initialized, Role: FOLLOWER
I20260927 00:52:48.844367 3497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dee74716694b1eaae8542d9324d34f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46359 } }
I20260927 00:52:48.844519 3497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:48.844587 3497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:48.844684 3497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:48.845598 3497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dee74716694b1eaae8542d9324d34f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46359 } }
I20260927 00:52:48.845908 3497 leader_election.cc:302] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81dee74716694b1eaae8542d9324d34f; no voters:
I20260927 00:52:48.846176 3497 leader_election.cc:288] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:48.846405 3502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:48.846750 3502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [term 1 LEADER]: Becoming Leader. State: Replica: 81dee74716694b1eaae8542d9324d34f, State: Running, Role: LEADER
I20260927 00:52:48.847159 3502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dee74716694b1eaae8542d9324d34f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46359 } }
I20260927 00:52:48.847383 3497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:48.848934 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81dee74716694b1eaae8542d9324d34f. Latest consensus state: current_term: 1 leader_uuid: "81dee74716694b1eaae8542d9324d34f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dee74716694b1eaae8542d9324d34f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46359 } } }
I20260927 00:52:48.849074 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:48.849261 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81dee74716694b1eaae8542d9324d34f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dee74716694b1eaae8542d9324d34f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46359 } } }
I20260927 00:52:48.849381 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:48.852684 3517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:48.852764 3517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:48.852838 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:48.853621 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:48.859889 3516 catalog_manager.cc:1347] Generated new cluster ID: f3a02e9357e94b3a952e2855beb0e4ad
I20260927 00:52:48.859987 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:48.868806 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:48.869954 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:48.876166 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81dee74716694b1eaae8542d9324d34f: Generated new TSK 0
I20260927 00:52:48.877213 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:48.910719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46359
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:49.043172 3523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:49.043552 3523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:49.043601 3523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:49.048098 3523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:49.048199 3523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:49.048293 3523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:49.054481 3523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:49.055997 3523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:49.057329 3523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:49.068619 3531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:49.068738 3528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:49.068883 3523 server_base.cc:1034] running on GCE node
W20260927 00:52:49.068738 3529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:49.069679 3523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:49.070540 3523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:49.071732 3523 hybrid_clock.cc:648] HybridClock initialized: now 1790470369071709 us; error 38 us; skew 500 ppm
Sep 27 00:52:49 dist-test-slave-9fh2 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:49.075001 3523 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:49.076138 3523 webserver.cc:460] Webserver started at http://127.0.98.1:35257/ using document root <none> and password file <none>
I20260927 00:52:49.076803 3523 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:49.076876 3523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:49.077116 3523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:49.079664 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "2373c05820cd47af803ea403e4efceac"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "b66f922f55d824d5747e147145523e1f"
server_key_iv: "fbd44affbdd4feede54d5e71640b6923"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.080308 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2373c05820cd47af803ea403e4efceac"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "b66f922f55d824d5747e147145523e1f"
server_key_iv: "fbd44affbdd4feede54d5e71640b6923"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.084998 3523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:49.088342 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.090072 3523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:52:49.090230 3523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "2373c05820cd47af803ea403e4efceac"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "b66f922f55d824d5747e147145523e1f"
server_key_iv: "fbd44affbdd4feede54d5e71640b6923"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.090387 3523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:49.146325 3523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:49.150512 3523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:49.150729 3523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:49.151521 3523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:49.152571 3523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:49.152627 3523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.152676 3523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:49.152709 3523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.164132 3523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44641
I20260927 00:52:49.164218 3651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44641 every 8 connection(s)
I20260927 00:52:49.165328 3523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:49.170054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3523
I20260927 00:52:49.170166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:49.170470 392 external_mini_cluster.cc:1428] Setting key 9c45b8057ff20eff5e543e5b6f781435
Sep 27 00:52:49 dist-test-slave-9fh2 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:49.182637 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:49.167652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55937 (local address 127.0.98.62:46359)
0927 00:52:49.167821 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:49.167825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:49.168896 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 00:52:49.170619 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:49.170622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:49.170683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:49.170757 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:49.172326 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.172986 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.174776 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.175125 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.175294 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:49.178547 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:49.178551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:49.178554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:49.178586 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:49.180186 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:49.180933 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:49.180937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:49.180938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:49.181045 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:49.181388 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:49.181391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:49.181392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:49.181644 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:49.181768 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:49.182328 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 00:52:49.182473 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:52:49.183528 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:49.168103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46359 (local address 127.0.98.1:55937)
0927 00:52:49.168709 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:49.168771 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:49.170341 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:49.170986 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:49.170994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:49.171517 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:49.172180 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:49.172195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.173155 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.173160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:49.174620 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:49.174637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.175391 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.175394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:49.175585 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:49.176272 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:49.176292 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:49.178281 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:49.180342 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:49.180350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:49.180369 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:49.180781 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:49.181141 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:49.181143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:49.181145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:49.181292 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:49.181739 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:49.181744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:49.182052 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 00:52:49.182234 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:52:49.182538 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":152,"threads_started":1}
I20260927 00:52:49.185053 3652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46359
I20260927 00:52:49.185487 3652 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:49.186164 3652 heartbeater.cc:511] Master 127.0.98.62:46359 requested a full tablet report, sending...
I20260927 00:52:49.188299 3461 ts_manager.cc:194] Registered new tserver with Master: 2373c05820cd47af803ea403e4efceac (127.0.98.1:44641)
I20260927 00:52:49.189607 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 00:52:49.506410 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.334s user 0.004s sys 0.001s
I20260927 00:52:49.507038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46359
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:49.635967 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:49.636341 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:49.636394 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:49.640834 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:49.640928 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:49.640996 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:49.647006 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:49.648469 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:49.649749 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:49.661397 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:49.661397 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:49.661449 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:49.662384 3657 server_base.cc:1034] running on GCE node
I20260927 00:52:49.663004 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:49.663939 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:49.665149 3657 hybrid_clock.cc:648] HybridClock initialized: now 1790470369665125 us; error 39 us; skew 500 ppm
Sep 27 00:52:49 dist-test-slave-9fh2 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:49.668694 3657 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:49.669874 3657 webserver.cc:460] Webserver started at http://127.0.98.2:38391/ using document root <none> and password file <none>
I20260927 00:52:49.670569 3657 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:49.670641 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:49.670848 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:49.673437 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "db4288b8bd92483db5c0950f89bfec5c"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "9bde582f9c348a1bec74fa12b8eb1aa9"
server_key_iv: "0b59e772578609896a6f736f9cae11e6"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.674108 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db4288b8bd92483db5c0950f89bfec5c"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "9bde582f9c348a1bec74fa12b8eb1aa9"
server_key_iv: "0b59e772578609896a6f736f9cae11e6"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.678874 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:52:49.682869 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.684677 3657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:49.684819 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "db4288b8bd92483db5c0950f89bfec5c"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "9bde582f9c348a1bec74fa12b8eb1aa9"
server_key_iv: "0b59e772578609896a6f736f9cae11e6"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.684954 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:49.703258 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:49.706899 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:49.707093 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:49.707836 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:49.708873 3657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:49.708932 3657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.708976 3657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:49.708990 3657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.718916 3657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40927
I20260927 00:52:49.719020 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40927 every 8 connection(s)
I20260927 00:52:49.719973 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:49.724972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3657
I20260927 00:52:49.725117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:49.725435 392 external_mini_cluster.cc:1428] Setting key b1f47205b61ea031c65ed03892c13083
Sep 27 00:52:49 dist-test-slave-9fh2 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:49.736361 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:49.722308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60261 (local address 127.0.98.62:46359)
0927 00:52:49.722699 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:49.722703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:49.723642 (+ 939us) server_negotiation.cc:407] Connection header received
0927 00:52:49.725195 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:49.725207 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:49.725269 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:49.725368 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:49.726794 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.727493 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.728662 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.728886 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.728981 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:49.732109 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:49.732112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:49.732114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:49.732138 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:49.733834 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:49.734405 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:49.734409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:49.734411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:49.734515 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:49.734988 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:49.734990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:49.734992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:49.735246 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:49.735352 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:49.736014 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 00:52:49.736150 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":154,"threads_started":1}
I20260927 00:52:49.737493 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:49.722817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46359 (local address 127.0.98.2:60261)
0927 00:52:49.723439 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:49.723507 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:49.724942 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:49.725584 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:49.725592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:49.726071 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:49.726643 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:49.726658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.727617 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.727620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:49.728538 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:49.728547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.729032 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.729036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:49.729196 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:49.729881 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:49.729898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:49.731870 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:49.733973 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:49.733979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:49.733995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:49.734299 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:49.734665 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:49.734669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:49.734670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:49.734858 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:49.735374 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:49.735380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:49.735740 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 00:52:49.735937 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:52:49.736311 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:52:49.739099 3786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46359
I20260927 00:52:49.739403 3786 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:49.739998 3786 heartbeater.cc:511] Master 127.0.98.62:46359 requested a full tablet report, sending...
I20260927 00:52:49.741235 3461 ts_manager.cc:194] Registered new tserver with Master: db4288b8bd92483db5c0950f89bfec5c (127.0.98.2:40927)
I20260927 00:52:49.741977 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:49.758713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46359
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:49.886713 3792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:49.887051 3792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:49.887099 3792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:49.891732 3792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:49.891829 3792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:49.891898 3792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:49.897994 3792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:49.899406 3792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:49.900632 3792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:52:49.912060 3792 server_base.cc:1034] running on GCE node
W20260927 00:52:49.911974 3800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:49.912012 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:49.911973 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:49.913098 3792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:49.914012 3792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:49.915324 3792 hybrid_clock.cc:648] HybridClock initialized: now 1790470369915291 us; error 49 us; skew 500 ppm
Sep 27 00:52:49 dist-test-slave-9fh2 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:49.918762 3792 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:49.920125 3792 webserver.cc:460] Webserver started at http://127.0.98.3:42983/ using document root <none> and password file <none>
I20260927 00:52:49.920849 3792 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:49.920930 3792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:49.921173 3792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:49.923767 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "b3917014edab44008f972ef7c5615c70"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "0fb174298bba2c041b7a7671ce3f164f"
server_key_iv: "8b322a6eb56b15841b829a13abd14e35"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.924434 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b3917014edab44008f972ef7c5615c70"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "0fb174298bba2c041b7a7671ce3f164f"
server_key_iv: "8b322a6eb56b15841b829a13abd14e35"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.929245 3792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:52:49.932497 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.934192 3792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:52:49.934420 3792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "b3917014edab44008f972ef7c5615c70"
format_stamp: "Formatted at 2026-09-27 00:52:49 on dist-test-slave-9fh2"
server_key: "0fb174298bba2c041b7a7671ce3f164f"
server_key_iv: "8b322a6eb56b15841b829a13abd14e35"
server_key_version: "encryptionkey@0"
I20260927 00:52:49.934633 3792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:49.969359 3792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:49.973373 3792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:49.973605 3792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:49.974344 3792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:49.975425 3792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:49.975479 3792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.975543 3792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:49.975567 3792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:49.986577 3792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39995
I20260927 00:52:49.986644 3920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39995 every 8 connection(s)
I20260927 00:52:49.987756 3792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:49.996862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3792
I20260927 00:52:49.997026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:49.997359 392 external_mini_cluster.cc:1428] Setting key 259b5e03a190062e31505c5be4153c65
Sep 27 00:52:49 dist-test-slave-9fh2 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:50.003842 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:49.990085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35593 (local address 127.0.98.62:46359)
0927 00:52:49.990245 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:49.990249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:49.991563 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 00:52:49.993157 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:49.993161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:49.993220 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:49.993292 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:49.994686 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.995313 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.996453 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.996702 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.996790 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:50.000042 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:50.000045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:50.000047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:50.000074 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:50.001446 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:50.002098 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:50.002100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:50.002102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:50.002181 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:50.002545 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:50.002548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:50.002549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:50.002731 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:50.002821 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:50.003581 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 00:52:50.003701 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:52:50.004963 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:49.990659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46359 (local address 127.0.98.3:35593)
0927 00:52:49.991371 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:49.991425 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:49.992901 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:49.993503 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:49.993512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:49.993955 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:49.994542 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:49.994556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.995435 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.995438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:49.996343 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:49.996352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:49.996847 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:49.996851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:49.997023 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:49.997819 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:49.997839 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:49.999825 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:50.001569 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:50.001575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:50.001590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:50.001936 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:50.002295 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:50.002298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:50.002299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:50.002451 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:50.002842 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:50.002849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:50.003228 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 00:52:50.003500 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 00:52:50.003852 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":196,"threads_started":1}
I20260927 00:52:50.006466 3921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46359
I20260927 00:52:50.006747 3921 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:50.007375 3921 heartbeater.cc:511] Master 127.0.98.62:46359 requested a full tablet report, sending...
I20260927 00:52:50.008612 3461 ts_manager.cc:194] Registered new tserver with Master: b3917014edab44008f972ef7c5615c70 (127.0.98.3:39995)
I20260927 00:52:50.009193 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35593
I20260927 00:52:50.012465 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:52:50 dist-test-slave-9fh2 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 00:52:50 dist-test-slave-9fh2 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470370, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:50.040400 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:50.031921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.0.98.62:46359)
0927 00:52:50.032056 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:50.032061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:50.032294 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:52:50.032435 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:50.032438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:50.032481 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:50.032568 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:50.033293 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:50.033816 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:50.034620 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:50.034842 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:50.034925 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:50.037407 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:50.037410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:50.037412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:50.037435 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:50.038737 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:50.039118 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:50.039121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:50.039122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:50.039240 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:50.039560 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:50.039563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:50.039565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:50.039783 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:50.039903 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:50.040121 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 00:52:50.040242 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 00:52:50.040797 3461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38080
I20260927 00:52:50.041823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3523
I20260927 00:52:50.048622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3657
I20260927 00:52:50.054286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3792
I20260927 00:52:50.059612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3429
2026-09-27T00:52:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2529 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:50 dist-test-slave-9fh2 krb5kdc[3933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:50 dist-test-slave-9fh2 krb5kdc[3933](info): set up 2 sockets
Sep 27 00:52:50 dist-test-slave-9fh2 krb5kdc[3933](info): commencing operation
krb5kdc: starting...
W20260927 00:52:51.027637 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:51.161041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37759
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:51.294056 3946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:51.294427 3946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:51.294474 3946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:51.298903 3946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:51.299000 3946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:51.299019 3946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:51.299036 3946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:51.299050 3946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:51.305104 3946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:51.306533 3946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:51.307812 3946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:52:51.317787 3946 server_base.cc:1034] running on GCE node
W20260927 00:52:51.317723 3951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:51.317723 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:51.317770 3954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:51.318948 3946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:51.320394 3946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:51.321611 3946 hybrid_clock.cc:648] HybridClock initialized: now 1790470371321575 us; error 54 us; skew 500 ppm
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:51.328909 3946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:51.330327 3946 webserver.cc:460] Webserver started at http://127.0.98.62:43051/ using document root <none> and password file <none>
I20260927 00:52:51.331049 3946 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:51.331127 3946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:51.331419 3946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:51.334009 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "a7159d5ee47a4a8db6b2d009196de6f0"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "7c04bcbefb796bd9cf9749878f661ebd"
server_key_iv: "ce23a8dd30f9a83b154afd0f39a74c17"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.334669 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7159d5ee47a4a8db6b2d009196de6f0"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "7c04bcbefb796bd9cf9749878f661ebd"
server_key_iv: "ce23a8dd30f9a83b154afd0f39a74c17"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.339507 3946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:52:51.342972 3961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.344748 3946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:51.344945 3946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "a7159d5ee47a4a8db6b2d009196de6f0"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "7c04bcbefb796bd9cf9749878f661ebd"
server_key_iv: "ce23a8dd30f9a83b154afd0f39a74c17"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.345098 3946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:51.367234 3946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:51.371094 3946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:51.371349 3946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:51.379971 3946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37759
I20260927 00:52:51.380005 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37759 every 8 connection(s)
I20260927 00:52:51.381170 3946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:51.384399 4014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:51.389345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 3946
I20260927 00:52:51.389526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:51.389839 392 external_mini_cluster.cc:1428] Setting key 562e9694d15341f3e5bd63ada54c3497
I20260927 00:52:51.390391 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0: Bootstrap starting.
I20260927 00:52:51.393173 4014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:51.394085 4014 log.cc:824] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:51.396747 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0: No bootstrap required, opened a new log
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:51.400899 4014 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37759 } }
I20260927 00:52:51.401144 4014 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:51.401185 4014 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7159d5ee47a4a8db6b2d009196de6f0, State: Initialized, Role: FOLLOWER
I20260927 00:52:51.401665 4014 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37759 } }
I20260927 00:52:51.401831 4014 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:51.401904 4014 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:51.402002 4014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:51.402921 4014 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37759 } }
I20260927 00:52:51.403304 4014 leader_election.cc:302] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7159d5ee47a4a8db6b2d009196de6f0; no voters:
I20260927 00:52:51.403615 4014 leader_election.cc:288] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:51.403731 4019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:51.403954 4019 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [term 1 LEADER]: Becoming Leader. State: Replica: a7159d5ee47a4a8db6b2d009196de6f0, State: Running, Role: LEADER
I20260927 00:52:51.404356 4019 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37759 } }
I20260927 00:52:51.404960 4014 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:51.405610 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7159d5ee47a4a8db6b2d009196de6f0. Latest consensus state: current_term: 1 leader_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37759 } } }
I20260927 00:52:51.406307 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:51.406191 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7159d5ee47a4a8db6b2d009196de6f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37759 } } }
I20260927 00:52:51.406602 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:51.407680 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:51.391690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.0.98.62:37759)
0927 00:52:51.392269 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:51.392289 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:51.392337 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:52:51.393218 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:51.393244 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:51.393706 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:51.394109 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:51.395103 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.396521 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.397598 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.398278 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.398442 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:51.401553 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:51.401561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:51.401584 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:51.401628 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:51.403772 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:51.404344 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:51.404351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:51.404358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:51.404519 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:51.404776 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:51.404779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:51.404781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:51.405363 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:51.405601 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:51.405891 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 00:52:51.406278 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":168,"threads_started":1}
I20260927 00:52:51.408239 4028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:51.411612 4028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:51.418576 4028 catalog_manager.cc:1347] Generated new cluster ID: cf4e5ba787064cd4b90aac416506d8f1
I20260927 00:52:51.418674 4028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:51.430150 4028 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:51.431216 4028 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:51.436728 4028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7159d5ee47a4a8db6b2d009196de6f0: Generated new TSK 0
I20260927 00:52:51.437444 4028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:51.474355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37759
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:51.610644 4040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:51.611013 4040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:51.611083 4040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:51.615473 4040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:51.615572 4040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:51.615664 4040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:51.621598 4040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:51.623188 4040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:51.624464 4040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:51.635759 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:51.635797 4040 server_base.cc:1034] running on GCE node
W20260927 00:52:51.635807 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:51.635833 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:51.636649 4040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:51.637422 4040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:51.638609 4040 hybrid_clock.cc:648] HybridClock initialized: now 1790470371638573 us; error 51 us; skew 500 ppm
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:51.641664 4040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:51.642679 4040 webserver.cc:460] Webserver started at http://127.0.98.1:44279/ using document root <none> and password file <none>
I20260927 00:52:51.643357 4040 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:51.643433 4040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:51.643640 4040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:51.645851 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "d116d3ceeefd437680cac0406a867aff"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "15d63d32790187895b1fdf45fa5ac8ca"
server_key_iv: "980a3da5ad1ce0a3bad5a48e4d9e6f55"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.646436 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d116d3ceeefd437680cac0406a867aff"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "15d63d32790187895b1fdf45fa5ac8ca"
server_key_iv: "980a3da5ad1ce0a3bad5a48e4d9e6f55"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.650876 4040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:52:51.653918 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.655514 4040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:52:51.655640 4040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "d116d3ceeefd437680cac0406a867aff"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "15d63d32790187895b1fdf45fa5ac8ca"
server_key_iv: "980a3da5ad1ce0a3bad5a48e4d9e6f55"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.655771 4040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:51.694823 4040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:51.698184 4040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:51.698369 4040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:51.698971 4040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:51.699930 4040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:51.699983 4040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.700038 4040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:51.700064 4040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.711901 4040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34103
I20260927 00:52:51.712033 4168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34103 every 8 connection(s)
I20260927 00:52:51.712940 4040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:51.723066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4040
I20260927 00:52:51.723234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:51.723536 392 external_mini_cluster.cc:1428] Setting key 3ffc1718532bada37135f56fd070e2e0
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:51.729033 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:51.715293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50153 (local address 127.0.98.62:37759)
0927 00:52:51.715475 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:51.715480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:51.716557 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 00:52:51.718210 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:51.718214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:51.718269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:51.718360 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:51.719905 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.720475 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.721788 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.722045 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.722139 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:51.725384 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:51.725388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:51.725390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:51.725415 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:51.726820 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:51.727449 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:51.727453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:51.727454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:51.727538 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:51.727894 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:51.727897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:51.727898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:51.728080 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:51.728161 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:51.728739 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 00:52:51.728864 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:52:51.729954 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:51.715759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37759 (local address 127.0.98.1:50153)
0927 00:52:51.716355 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:51.716415 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:51.717966 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:51.718565 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:51.718573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:51.719035 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:51.719713 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:51.719733 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.720627 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.720630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:51.721651 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:51.721661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.722245 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.722249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:51.722426 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:51.723247 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:51.723275 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:51.725186 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:51.726947 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:51.726954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:51.726971 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:51.727346 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:51.727622 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:51.727625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:51.727627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:51.727781 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:51.728178 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:51.728185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:51.728479 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 00:52:51.728659 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:52:51.729026 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":132,"threads_started":1}
I20260927 00:52:51.731465 4169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37759
I20260927 00:52:51.731715 4169 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:51.732225 4169 heartbeater.cc:511] Master 127.0.98.62:37759 requested a full tablet report, sending...
I20260927 00:52:51.733851 3978 ts_manager.cc:194] Registered new tserver with Master: d116d3ceeefd437680cac0406a867aff (127.0.98.1:34103)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:51.735020 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:51.755303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37759
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:51.881716 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:51.882076 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:51.882130 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:51.886487 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:51.886565 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:51.886637 4174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:51.892619 4174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:51.894038 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:51.895355 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:51.906548 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:51.906558 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:51.907094 4174 server_base.cc:1034] running on GCE node
W20260927 00:52:51.906548 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:51.907749 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:51.908449 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:51.909627 4174 hybrid_clock.cc:648] HybridClock initialized: now 1790470371909575 us; error 73 us; skew 500 ppm
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:51.912737 4174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:51.913712 4174 webserver.cc:460] Webserver started at http://127.0.98.2:41289/ using document root <none> and password file <none>
I20260927 00:52:51.914322 4174 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:51.914389 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:51.914582 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:51.916738 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "d1447978880948c8bbcfd70af1022ee1"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "00bb6d3ff92d7a90a94c526a755f54e4"
server_key_iv: "8054e696bfd4e0ca6947ee50bccfe470"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.917306 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d1447978880948c8bbcfd70af1022ee1"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "00bb6d3ff92d7a90a94c526a755f54e4"
server_key_iv: "8054e696bfd4e0ca6947ee50bccfe470"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.921613 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:52:51.924521 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.926055 4174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:51.926193 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "d1447978880948c8bbcfd70af1022ee1"
format_stamp: "Formatted at 2026-09-27 00:52:51 on dist-test-slave-9fh2"
server_key: "00bb6d3ff92d7a90a94c526a755f54e4"
server_key_iv: "8054e696bfd4e0ca6947ee50bccfe470"
server_key_version: "encryptionkey@0"
I20260927 00:52:51.926331 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:51.941272 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:51.944486 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:51.944662 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:51.945261 4174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:51.946094 4174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:51.946144 4174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.946199 4174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:51.946233 4174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:51.955910 4174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39769
I20260927 00:52:51.956009 4302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39769 every 8 connection(s)
I20260927 00:52:51.956844 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:51.962883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4174
I20260927 00:52:51.963024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:51.963353 392 external_mini_cluster.cc:1428] Setting key 2a914715d30750ba836678405f757ece
Sep 27 00:52:51 dist-test-slave-9fh2 krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:51.972769 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:51.959090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42309 (local address 127.0.98.62:37759)
0927 00:52:51.959289 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:51.959294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:51.960400 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 00:52:51.961954 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:51.961957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:51.962012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:51.962085 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:51.963370 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.963982 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.965112 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.965330 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.965415 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:51.968401 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:51.968406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:51.968408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:51.968443 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:51.970138 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:51.970822 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:51.970825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:51.970827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:51.970920 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:51.971344 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:51.971347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:51.971348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:51.971581 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:51.971661 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:51.972454 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 00:52:51.972597 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:51.973839 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:51.959652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37759 (local address 127.0.98.2:42309)
0927 00:52:51.960220 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:51.960277 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:51.961734 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:51.962275 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:51.962283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:51.962655 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:51.963237 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:51.963254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.964109 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.964111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:51.964993 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:51.965002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:51.965473 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:51.965477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:51.965636 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:51.966313 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:51.966330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:51.968153 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:51.970290 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:51.970297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:51.970315 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:51.970690 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:51.971015 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:51.971018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:51.971020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:51.971235 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:51.971691 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:51.971698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:51.972057 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 00:52:51.972289 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 00:52:51.972669 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":156,"threads_started":1}
I20260927 00:52:51.975409 4303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37759
I20260927 00:52:51.975752 4303 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:52:51.976487 4303 heartbeater.cc:511] Master 127.0.98.62:37759 requested a full tablet report, sending...
I20260927 00:52:51.977984 3978 ts_manager.cc:194] Registered new tserver with Master: d1447978880948c8bbcfd70af1022ee1 (127.0.98.2:39769)
I20260927 00:52:51.978679 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:51.995031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37759
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:52.125068 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:52.125432 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:52.125481 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:52.129812 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:52.129894 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:52.129961 4308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:52.135838 4308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:52.137230 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:52.138468 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:52.149657 4313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:52.149682 4314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:52.149959 4316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:52.150053 4308 server_base.cc:1034] running on GCE node
I20260927 00:52:52.150704 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:52.151566 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:52.152753 4308 hybrid_clock.cc:648] HybridClock initialized: now 1790470372152728 us; error 38 us; skew 500 ppm
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:52.156054 4308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:52.157253 4308 webserver.cc:460] Webserver started at http://127.0.98.3:40235/ using document root <none> and password file <none>
I20260927 00:52:52.158005 4308 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:52.158098 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:52.158329 4308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:52.161175 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b3d84b7a94e41ee84c7ea90cf2592ad"
format_stamp: "Formatted at 2026-09-27 00:52:52 on dist-test-slave-9fh2"
server_key: "7e5c7726b78a90834d216f4ef245cbdb"
server_key_iv: "ccc52454fbda1198e681022bf38f5450"
server_key_version: "encryptionkey@0"
I20260927 00:52:52.162102 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b3d84b7a94e41ee84c7ea90cf2592ad"
format_stamp: "Formatted at 2026-09-27 00:52:52 on dist-test-slave-9fh2"
server_key: "7e5c7726b78a90834d216f4ef245cbdb"
server_key_iv: "ccc52454fbda1198e681022bf38f5450"
server_key_version: "encryptionkey@0"
I20260927 00:52:52.167582 4308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:52:52.171049 4323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:52.172855 4308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:52.173032 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "3b3d84b7a94e41ee84c7ea90cf2592ad"
format_stamp: "Formatted at 2026-09-27 00:52:52 on dist-test-slave-9fh2"
server_key: "7e5c7726b78a90834d216f4ef245cbdb"
server_key_iv: "ccc52454fbda1198e681022bf38f5450"
server_key_version: "encryptionkey@0"
I20260927 00:52:52.173173 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:52.203675 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:52.207763 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:52.207973 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:52.208650 4308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:52.209890 4308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:52.209944 4308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:52.209991 4308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:52.210021 4308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:52.220729 4308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33391
I20260927 00:52:52.220793 4436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33391 every 8 connection(s)
I20260927 00:52:52.221895 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:52.223097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4308
I20260927 00:52:52.223253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:52.223591 392 external_mini_cluster.cc:1428] Setting key 54765d0c9da0baa9670b4564d86fe1f1
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:52.239019 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:52.224320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54021 (local address 127.0.98.62:37759)
0927 00:52:52.224455 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:52.224459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:52.225754 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 00:52:52.227404 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:52.227408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:52.227476 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:52.227564 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:52.228855 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.229476 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.230868 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.231139 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.231307 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:52.234093 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:52.234097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:52.234099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:52.234134 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:52.235924 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:52.236584 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:52.236588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:52.236590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:52.236717 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:52.237324 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:52.237330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:52.237331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:52.237596 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:52.237716 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:52.238635 (+ 919us) server_negotiation.cc:299] Negotiation successful
0927 00:52:52.238827 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:52:52.240151 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:52.224946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37759 (local address 127.0.98.3:54021)
0927 00:52:52.225563 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:52.225633 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:52.227102 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:52.227726 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:52.227733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:52.228150 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:52.228694 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:52.228707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.229650 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.229654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:52.230732 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:52.230744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.231380 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.231384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:52.231553 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:52.232274 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:52.232292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:52.233853 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:52.236084 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:52.236091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:52.236113 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:52.236446 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:52.236881 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:52.236885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:52.236887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:52.237186 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:52.237758 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:52.237768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:52.238316 (+ 548us) client_negotiation.cc:769] Sending connection context
0927 00:52:52.238569 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 00:52:52.239027 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":160,"threads_started":1}
I20260927 00:52:52.241775 4437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37759
I20260927 00:52:52.242116 4437 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:52.242791 4437 heartbeater.cc:511] Master 127.0.98.62:37759 requested a full tablet report, sending...
I20260927 00:52:52.244268 3978 ts_manager.cc:194] Registered new tserver with Master: 3b3d84b7a94e41ee84c7ea90cf2592ad (127.0.98.3:33391)
I20260927 00:52:52.245003 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54021
I20260927 00:52:52.252004 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:52.282148 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:52.270699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.0.98.1:34103)
0927 00:52:52.271059 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:52.271070 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:52.271096 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:52:52.271187 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:52.271195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:52.271463 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:52.271623 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:52.272494 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.273331 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.274199 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.274647 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.274764 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:52.277698 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:52.277710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:52.277730 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:52.277782 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:52.279730 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:52.280460 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:52.280467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:52.280468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:52.280584 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:52.280906 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:52.280908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:52.280910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:52.281279 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:52.281537 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:52.281735 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:52:52.281926 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:52.294865 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:52.286085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.98.62:37759)
0927 00:52:52.286411 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:52.286416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:52.286437 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:52:52.286493 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:52.286496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:52.286552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:52.286658 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:52.287390 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.287999 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.288843 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:52.289072 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:52.289161 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:52.291794 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:52.291798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:52.291800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:52.291832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:52.293254 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:52.293621 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:52.293624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:52.293625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:52.293723 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:52.293940 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:52.293942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:52.293944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:52.294209 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:52.294440 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:52.294587 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:52:52.294707 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260927 00:52:52.295426 3978 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418
I20260927 00:52:52.296176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4040
I20260927 00:52:52.301617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4174
I20260927 00:52:52.307827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4308
I20260927 00:52:52.313548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 3946
2026-09-27T00:52:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2253 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[4449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[4449](info): set up 2 sockets
Sep 27 00:52:52 dist-test-slave-9fh2 krb5kdc[4449](info): commencing operation
krb5kdc: starting...
W20260927 00:52:53.275549 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:53 dist-test-slave-9fh2 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:53.408766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:53.526149 4462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:53.526417 4462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:53.526458 4462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:53.530548 4462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:53.530623 4462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:53.530642 4462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:53.530658 4462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:53.530673 4462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:53.536298 4462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:53.537449 4462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:53.538486 4462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:53.548404 4468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:53.548378 4467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:53.548549 4470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:53.549074 4462 server_base.cc:1034] running on GCE node
I20260927 00:52:53.549903 4462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:53.551667 4462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:53.552892 4462 hybrid_clock.cc:648] HybridClock initialized: now 1790470373552859 us; error 54 us; skew 500 ppm
Sep 27 00:52:53 dist-test-slave-9fh2 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:53.556861 4462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:53.557847 4462 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:53.557924 4462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:53.558159 4462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:53.560753 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "09bf9b17f83b4b09b39b1fd2ae581bff"
format_stamp: "Formatted at 2026-09-27 00:52:53 on dist-test-slave-9fh2"
server_key: "ed0a73e5ffeebfd401a550ab30a8ce74"
server_key_iv: "7f950d4e61f5e0e4709282d39bf62a26"
server_key_version: "encryptionkey@0"
I20260927 00:52:53.561379 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "09bf9b17f83b4b09b39b1fd2ae581bff"
format_stamp: "Formatted at 2026-09-27 00:52:53 on dist-test-slave-9fh2"
server_key: "ed0a73e5ffeebfd401a550ab30a8ce74"
server_key_iv: "7f950d4e61f5e0e4709282d39bf62a26"
server_key_version: "encryptionkey@0"
I20260927 00:52:53.566207 4462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:52:53.569551 4476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:53.571496 4462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:53.571681 4462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "09bf9b17f83b4b09b39b1fd2ae581bff"
format_stamp: "Formatted at 2026-09-27 00:52:53 on dist-test-slave-9fh2"
server_key: "ed0a73e5ffeebfd401a550ab30a8ce74"
server_key_iv: "7f950d4e61f5e0e4709282d39bf62a26"
server_key_version: "encryptionkey@0"
I20260927 00:52:53.571831 4462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:53.587980 4462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:53.591908 4462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:53.592124 4462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:53.600344 4462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41317
I20260927 00:52:53.600617 4528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41317 every 8 connection(s)
I20260927 00:52:53.601140 4462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:53.604168 4529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:53.607087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4462
I20260927 00:52:53.607247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:53.607559 392 external_mini_cluster.cc:1428] Setting key c72059cfd5c495fe2b8f7a811a82e45e
I20260927 00:52:53.610195 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: Bootstrap starting.
I20260927 00:52:53.612754 4529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:53.613463 4529 log.cc:824] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:53.615432 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: No bootstrap required, opened a new log
Sep 27 00:52:53 dist-test-slave-9fh2 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:53.619498 4529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260927 00:52:53.619776 4529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:53.619817 4529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bf9b17f83b4b09b39b1fd2ae581bff, State: Initialized, Role: FOLLOWER
I20260927 00:52:53.620306 4529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260927 00:52:53.620460 4529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:53.620538 4529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:53.620639 4529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:53.621628 4529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260927 00:52:53.621992 4529 leader_election.cc:302] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bf9b17f83b4b09b39b1fd2ae581bff; no voters:
I20260927 00:52:53.622363 4529 leader_election.cc:288] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:53.622450 4534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:53.622656 4534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [term 1 LEADER]: Becoming Leader. State: Replica: 09bf9b17f83b4b09b39b1fd2ae581bff, State: Running, Role: LEADER
I20260927 00:52:53.623063 4534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260927 00:52:53.623720 4529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:53.624878 4535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } }
I20260927 00:52:53.625067 4535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:53.625190 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:53.609267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.0.98.62:41317)
0927 00:52:53.609763 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:53.609783 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:53.609830 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:52:53.610676 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:53.610698 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:53.611138 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:53.611564 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:53.612485 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:53.613780 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:53.614756 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:53.615360 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:53.615535 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:53.618573 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:53.618582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:53.618601 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:53.618641 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:53.620885 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:53.621549 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:53.621556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:53.621562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:53.621712 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:53.622148 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:53.622154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:53.622156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:53.622936 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:53.623217 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:53.623583 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 00:52:53.623988 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
I20260927 00:52:53.626170 4536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bf9b17f83b4b09b39b1fd2ae581bff. Latest consensus state: current_term: 1 leader_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf9b17f83b4b09b39b1fd2ae581bff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } }
I20260927 00:52:53.626286 4536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:53.628886 4549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:53.628957 4549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:53.629043 4544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:53.629760 4544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:53.635900 4544 catalog_manager.cc:1347] Generated new cluster ID: afd393603c3e4983abeea58b5c864261
I20260927 00:52:53.635979 4544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:53.657233 4544 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:53.658880 4544 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:53.671209 4544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: Generated new TSK 0
I20260927 00:52:53.672045 4544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:53.702225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41317
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:53.811479 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:53.811746 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:53.811805 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:53.815799 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:53.815865 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:53.815951 4555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:53.821194 4555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:53.822225 4555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:53.823216 4555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:53.833489 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:53.833500 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:53.833561 4555 server_base.cc:1034] running on GCE node
W20260927 00:52:53.833515 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:53.834241 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:53.834918 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:53.836081 4555 hybrid_clock.cc:648] HybridClock initialized: now 1790470373836063 us; error 65 us; skew 500 ppm
Sep 27 00:52:53 dist-test-slave-9fh2 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:53.838757 4555 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:53.839691 4555 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:53.839783 4555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:53.839985 4555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:53.841979 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c59e1a935f64d6ca94e833af794dd0b"
format_stamp: "Formatted at 2026-09-27 00:52:53 on dist-test-slave-9fh2"
server_key: "a79f442fcf10ad9cba8d4309e8555229"
server_key_iv: "12acb1e6c513b5aea5fab16e3fb5fe3d"
server_key_version: "encryptionkey@0"
I20260927 00:52:53.842499 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c59e1a935f64d6ca94e833af794dd0b"
format_stamp: "Formatted at 2026-09-27 00:52:53 on dist-test-slave-9fh2"
server_key: "a79f442fcf10ad9cba8d4309e8555229"
server_key_iv: "12acb1e6c513b5aea5fab16e3fb5fe3d"
server_key_version: "encryptionkey@0"
I20260927 00:52:53.846680 4555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:52:53.849591 4569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:53.851255 4555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:52:53.851387 4555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "8c59e1a935f64d6ca94e833af794dd0b"
format_stamp: "Formatted at 2026-09-27 00:52:53 on dist-test-slave-9fh2"
server_key: "a79f442fcf10ad9cba8d4309e8555229"
server_key_iv: "12acb1e6c513b5aea5fab16e3fb5fe3d"
server_key_version: "encryptionkey@0"
I20260927 00:52:53.851503 4555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:53.877529 4555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:53.881446 4555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:53.881668 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:53.882391 4555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:53.883445 4555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:53.883512 4555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:53.883567 4555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:53.883599 4555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:53.894626 4555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41123
I20260927 00:52:53.894706 4682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41123 every 8 connection(s)
I20260927 00:52:53.895494 4555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:53.900239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4555
I20260927 00:52:53.900455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:53.900782 392 external_mini_cluster.cc:1428] Setting key 8db56e05e53a87b690a76923c27f7803
Sep 27 00:52:53 dist-test-slave-9fh2 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:53.911593 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:53.897863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41683 (local address 127.0.98.62:41317)
0927 00:52:53.898015 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:53.898019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:53.899274 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 00:52:53.900785 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:53.900788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:53.900843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:53.900923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:53.902447 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:53.903000 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:53.904210 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:53.904433 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:53.904523 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:53.907412 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:53.907417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:53.907419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:53.907456 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:53.908897 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:53.909648 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:53.909652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:53.909654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:53.909781 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:53.910203 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:53.910206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:53.910207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:53.910488 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:53.910573 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:53.911271 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 00:52:53.911441 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:52:53.912422 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:53.898370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41317 (local address 127.0.98.1:41683)
0927 00:52:53.899014 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:53.899119 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:53.900568 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:53.901222 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:53.901230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:53.901735 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:53.902310 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:53.902330 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:53.903125 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:53.903128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:53.904086 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:53.904096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:53.904583 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:53.904587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:53.904751 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:53.905333 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:53.905350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:53.907140 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:53.909034 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:53.909041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:53.909058 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:53.909421 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:53.909893 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:53.909897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:53.909899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:53.910079 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:53.910610 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:53.910616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:53.910966 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 00:52:53.911220 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 00:52:53.911500 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":169,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:52:53.914223 4683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41317
I20260927 00:52:53.914498 4683 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:53.915060 4683 heartbeater.cc:511] Master 127.0.98.62:41317 requested a full tablet report, sending...
I20260927 00:52:53.917065 4493 ts_manager.cc:194] Registered new tserver with Master: 8c59e1a935f64d6ca94e833af794dd0b (127.0.98.1:41123)
I20260927 00:52:53.918411 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:53.930926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41317
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:54.042332 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:54.042610 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:54.042656 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:54.046677 4688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:54.046747 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:54.046809 4688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:54.052098 4688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:54.053124 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:54.054095 4688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:54.064220 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:54.064174 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:54.064174 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:54.064531 4688 server_base.cc:1034] running on GCE node
I20260927 00:52:54.065073 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:54.065884 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:54.067067 4688 hybrid_clock.cc:648] HybridClock initialized: now 1790470374067044 us; error 45 us; skew 500 ppm
Sep 27 00:52:54 dist-test-slave-9fh2 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:54.070598 4688 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:54.071623 4688 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:54.071707 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:54.071990 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:54.074616 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb7d10c269984e88803aa684a5dd83f0"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "d4e9fe0e529943cffcefc4e361bc0d26"
server_key_iv: "cb3c8886d352bc99a437f0ed3652889a"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.075321 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb7d10c269984e88803aa684a5dd83f0"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "d4e9fe0e529943cffcefc4e361bc0d26"
server_key_iv: "cb3c8886d352bc99a437f0ed3652889a"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.080209 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:52:54.083173 4702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:54.084957 4688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:52:54.085115 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "bb7d10c269984e88803aa684a5dd83f0"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "d4e9fe0e529943cffcefc4e361bc0d26"
server_key_iv: "cb3c8886d352bc99a437f0ed3652889a"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.085258 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:54.101814 4688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:54.105846 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:54.106076 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:54.106837 4688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:54.107966 4688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:54.108018 4688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:54.108079 4688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:54.108104 4688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:54.118947 4688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45075
I20260927 00:52:54.118975 4815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45075 every 8 connection(s)
I20260927 00:52:54.119727 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:54.128029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4688
I20260927 00:52:54.128139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:54.128419 392 external_mini_cluster.cc:1428] Setting key fec3d42478b369e5d6c5eec94b96270c
Sep 27 00:52:54 dist-test-slave-9fh2 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:54.136351 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.121719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57367 (local address 127.0.98.62:41317)
0927 00:52:54.121901 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.121906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.122835 (+ 929us) server_negotiation.cc:407] Connection header received
0927 00:52:54.124646 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.124650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.124708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.124783 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:54.126358 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.126903 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.128471 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.128797 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.128895 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.132332 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:54.132335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:54.132337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:54.132361 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:54.133715 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:54.134391 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:54.134394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:54.134395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:54.134478 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:54.134896 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:54.134898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:54.134899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:54.135080 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:54.135197 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.136080 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.136217 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:52:54.137784 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.122060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41317 (local address 127.0.98.2:57367)
0927 00:52:54.122642 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.122696 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.124333 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.125069 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.125078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.125540 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:54.126205 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.126224 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.127063 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.127067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.128303 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.128339 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.128976 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.128980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.129178 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.130017 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:54.130044 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:54.132132 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:54.133844 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:54.133852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:54.133871 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:54.134285 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:54.134581 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:54.134584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:54.134586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:54.134792 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:54.135208 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:54.135216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:54.135650 (+ 434us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.135916 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.136411 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:54.139343 4816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41317
I20260927 00:52:54.139674 4816 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:54.140223 4816 heartbeater.cc:511] Master 127.0.98.62:41317 requested a full tablet report, sending...
I20260927 00:52:54.141366 4493 ts_manager.cc:194] Registered new tserver with Master: bb7d10c269984e88803aa684a5dd83f0 (127.0.98.2:45075)
I20260927 00:52:54.141957 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:54.159452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41317
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:54.293004 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:54.293365 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:54.293419 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:54.297879 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:54.297978 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:54.298048 4821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:54.304100 4821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:54.305470 4821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:54.306699 4821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:54.317920 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:54.317910 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:54.317901 4827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:54.318192 4821 server_base.cc:1034] running on GCE node
I20260927 00:52:54.318867 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:54.319641 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:54.320802 4821 hybrid_clock.cc:648] HybridClock initialized: now 1790470374320777 us; error 46 us; skew 500 ppm
Sep 27 00:52:54 dist-test-slave-9fh2 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:54.323728 4821 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:54.324616 4821 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:54.324692 4821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:54.324872 4821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:54.327118 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "55cd01ae4d8b25aec91c1f98c64620bc"
server_key_iv: "c46034b8c5b2c59dc2ae10c0d9ce01b6"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.327688 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "55cd01ae4d8b25aec91c1f98c64620bc"
server_key_iv: "c46034b8c5b2c59dc2ae10c0d9ce01b6"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.331799 4821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:54.334438 4835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:54.335976 4821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:54.336092 4821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "55cd01ae4d8b25aec91c1f98c64620bc"
server_key_iv: "c46034b8c5b2c59dc2ae10c0d9ce01b6"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.336189 4821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:54.350649 4821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:54.353965 4821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:54.354133 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:54.354699 4821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:54.355660 4821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:54.355711 4821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:54.355767 4821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:54.355845 4821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:54.365530 4821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46639
I20260927 00:52:54.365572 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46639 every 8 connection(s)
I20260927 00:52:54.366156 4821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:54.367899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4821
I20260927 00:52:54.368026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:54.368346 392 external_mini_cluster.cc:1428] Setting key 7fe72b8467a10f84e33635b2ec6c0a96
Sep 27 00:52:54 dist-test-slave-9fh2 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:54.382282 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.368544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60351 (local address 127.0.98.62:41317)
0927 00:52:54.368707 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.368713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.369604 (+ 891us) server_negotiation.cc:407] Connection header received
0927 00:52:54.371046 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.371049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.371106 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.371219 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:54.372437 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.373049 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.374241 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.374475 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.374557 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.377683 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:54.377688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:54.377690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:54.377727 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:54.379321 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:54.380109 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:54.380112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:54.380113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:54.380203 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:54.380705 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:54.380707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:54.380708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:54.380934 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:54.381075 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.381904 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.382115 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:52:54.383466 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.368895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41317 (local address 127.0.98.3:60351)
0927 00:52:54.369443 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.369497 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.370826 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.371386 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.371393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.371766 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:54.372284 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.372298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.373200 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.373203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.374131 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.374141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.374642 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.374645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.374797 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.375404 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:54.375421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:54.377407 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:54.379490 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:54.379498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:54.379517 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:54.379918 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:54.380402 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:54.380405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:54.380406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:54.380598 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:54.381065 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:54.381074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:54.381578 (+ 504us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.381838 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.382295 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":138,"threads_started":1}
I20260927 00:52:54.384776 4949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41317
I20260927 00:52:54.385049 4949 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:54.385656 4949 heartbeater.cc:511] Master 127.0.98.62:41317 requested a full tablet report, sending...
I20260927 00:52:54.386893 4493 ts_manager.cc:194] Registered new tserver with Master: 926d0f3bc1bd4b2083b5ddcbcbe99fb6 (127.0.98.3:46639)
I20260927 00:52:54.387543 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60351
I20260927 00:52:54.395570 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:52:54.405586 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.398057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.0.98.62:41317)
0927 00:52:54.398223 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.398228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.398377 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:52:54.398543 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.398546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.398592 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.398659 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:54.399531 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.400059 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.400895 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.401105 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.401188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.402580 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:54.402583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:54.402585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:54.402606 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:54.404054 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:54.404437 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:54.404441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:54.404443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:54.404526 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:54.404764 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:54.404768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:54.404770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:54.404985 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:54.405102 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.405313 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.405422 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:52:54.409286 4493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:54.413033 4493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:54.430517 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.423596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41123 (local address 127.0.0.1:47328)
0927 00:52:54.424640 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.424709 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.424898 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.426137 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.426141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.426160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:54.426246 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.426251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.427635 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.427638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.428796 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.428805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.429771 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.429775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.429932 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.429991 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.430119 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.430327 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":69,"thread_start_us":122,"threads_started":1}
I20260927 00:52:54.431380 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.423795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45075 (local address 127.0.0.1:45298)
0927 00:52:54.425782 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.425796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.425898 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.426650 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.426653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.426669 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:54.426765 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.426771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.427966 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.427970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.429329 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.429336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.430965 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.430968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.431050 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.431069 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.431118 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.431211 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1914,"mutex_wait_us":101,"thread_start_us":91,"threads_started":1}
I20260927 00:52:54.431473 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.424550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.0.98.3:46639)
0927 00:52:54.425221 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.425232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.425253 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:52:54.425323 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.425327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.425519 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.425699 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:54.426017 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.427781 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.429484 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.430514 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.430624 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.430765 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.431235 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.431339 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":100,"threads_started":1}
I20260927 00:52:54.431492 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.424550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.98.2:45075)
0927 00:52:54.425023 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.425035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.425900 (+ 865us) server_negotiation.cc:407] Connection header received
0927 00:52:54.426098 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.426104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.426334 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.426488 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:54.426871 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.427841 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.429493 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.430842 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.430960 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.431117 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.431203 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.431313 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
I20260927 00:52:54.431715 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.423632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.0.1:39168)
0927 00:52:54.425045 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.425060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.425157 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.425727 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.425732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.425762 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:54.425897 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.425906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.427924 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.427929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.429313 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.429327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.430829 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.430833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.430942 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.430970 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.431050 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.431597 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":41,"thread_start_us":117,"threads_started":1}
I20260927 00:52:54.432044 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.424597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.0.98.1:41123)
0927 00:52:54.425141 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.425150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.425175 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:52:54.425265 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.425272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.425519 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.426142 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:54.426372 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.427506 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.428949 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.429635 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.429754 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.429907 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.431301 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.431428 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":118,"threads_started":1}
I20260927 00:52:54.434837 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 00a00ba34d944819a013e70a9ce9dc4f (DEFAULT_TABLE table=test-table [id=9848f4fcb65942c2965e5e9ffbdfd45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:54.434832 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 00a00ba34d944819a013e70a9ce9dc4f (DEFAULT_TABLE table=test-table [id=9848f4fcb65942c2965e5e9ffbdfd45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:54.434939 4883 tablet_service.cc:1467] Processing CreateTablet for tablet 00a00ba34d944819a013e70a9ce9dc4f (DEFAULT_TABLE table=test-table [id=9848f4fcb65942c2965e5e9ffbdfd45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:54.436070 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a00ba34d944819a013e70a9ce9dc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:54.436070 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a00ba34d944819a013e70a9ce9dc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:54.436203 4883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a00ba34d944819a013e70a9ce9dc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:54.443586 4966 tablet_bootstrap.cc:492] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Bootstrap starting.
I20260927 00:52:54.444794 4967 tablet_bootstrap.cc:492] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: Bootstrap starting.
I20260927 00:52:54.446385 4966 tablet_bootstrap.cc:654] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:54.446988 4967 tablet_bootstrap.cc:654] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:54.447602 4968 tablet_bootstrap.cc:492] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Bootstrap starting.
I20260927 00:52:54.447768 4966 log.cc:824] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:54.447767 4967 log.cc:824] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:54.449579 4967 tablet_bootstrap.cc:492] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: No bootstrap required, opened a new log
I20260927 00:52:54.449723 4968 tablet_bootstrap.cc:654] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:54.449760 4967 ts_tablet_manager.cc:1397] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:52:54.450845 4968 log.cc:824] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:54.451365 4966 tablet_bootstrap.cc:492] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: No bootstrap required, opened a new log
I20260927 00:52:54.451602 4966 ts_tablet_manager.cc:1397] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:52:54.453166 4968 tablet_bootstrap.cc:492] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: No bootstrap required, opened a new log
I20260927 00:52:54.453377 4968 ts_tablet_manager.cc:1397] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:52:54.454201 4967 raft_consensus.cc:348] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.454502 4967 raft_consensus.cc:374] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:54.454566 4967 raft_consensus.cc:729] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c59e1a935f64d6ca94e833af794dd0b, State: Initialized, Role: FOLLOWER
I20260927 00:52:54.455062 4967 consensus_queue.cc:260] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.455416 4966 raft_consensus.cc:348] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.455700 4966 raft_consensus.cc:374] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:54.455758 4966 raft_consensus.cc:729] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926d0f3bc1bd4b2083b5ddcbcbe99fb6, State: Initialized, Role: FOLLOWER
I20260927 00:52:54.456142 4683 heartbeater.cc:503] Master 127.0.98.62:41317 was elected leader, sending a full tablet report...
I20260927 00:52:54.456249 4966 consensus_queue.cc:260] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.457211 4966 ts_tablet_manager.cc:1428] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:52:54.457253 4968 raft_consensus.cc:348] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.457572 4968 raft_consensus.cc:374] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:54.457625 4968 raft_consensus.cc:729] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7d10c269984e88803aa684a5dd83f0, State: Initialized, Role: FOLLOWER
I20260927 00:52:54.457671 4949 heartbeater.cc:503] Master 127.0.98.62:41317 was elected leader, sending a full tablet report...
I20260927 00:52:54.458097 4968 consensus_queue.cc:260] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.458868 4816 heartbeater.cc:503] Master 127.0.98.62:41317 was elected leader, sending a full tablet report...
I20260927 00:52:54.459404 4968 ts_tablet_manager.cc:1428] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
I20260927 00:52:54.460084 4967 ts_tablet_manager.cc:1428] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260927 00:52:54.596871 4972 raft_consensus.cc:482] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:54.597183 4972 raft_consensus.cc:504] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.598610 4972 leader_election.cc:288] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926d0f3bc1bd4b2083b5ddcbcbe99fb6 (127.0.98.3:46639), bb7d10c269984e88803aa684a5dd83f0 (127.0.98.2:45075)
I20260927 00:52:54.603638 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.598983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45075 (local address 127.0.98.1:49211)
0927 00:52:54.599420 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.599436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.599603 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.600228 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.600231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.600251 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:54.600351 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.600357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.601329 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.601332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.602426 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.602433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.603119 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.603122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.603256 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.603281 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.603362 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.603433 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":61824,"thread_start_us":184,"threads_started":1}
I20260927 00:52:54.603650 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.599119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49211 (local address 127.0.98.2:45075)
0927 00:52:54.599285 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.599289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.599594 (+ 305us) server_negotiation.cc:407] Connection header received
0927 00:52:54.600026 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.600029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.600081 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.600151 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:54.600464 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.601204 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.602559 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.603000 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.603106 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.603179 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.603401 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.603506 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:52:54.603732 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.599039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42689 (local address 127.0.98.3:46639)
0927 00:52:54.599218 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.599222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.599555 (+ 333us) server_negotiation.cc:407] Connection header received
0927 00:52:54.599702 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.599705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.599779 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.599883 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:52:54.600261 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.600848 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.602346 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.603034 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.603165 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.603259 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.603503 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.603611 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:52:54.604077 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.598926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46639 (local address 127.0.98.1:42689)
0927 00:52:54.599420 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.599436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:54.599589 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:54.599885 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:54.599887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:54.599904 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:52:54.600031 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.600035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.600961 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.600963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.602193 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:54.602203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.603270 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.603273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:54.603366 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.603385 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:52:54.603857 (+ 472us) client_negotiation.cc:240] Negotiation successful
0927 00:52:54.603930 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":128,"threads_started":1}
I20260927 00:52:54.604521 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a00ba34d944819a013e70a9ce9dc4f" candidate_uuid: "8c59e1a935f64d6ca94e833af794dd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7d10c269984e88803aa684a5dd83f0" is_pre_election: true
I20260927 00:52:54.604753 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a00ba34d944819a013e70a9ce9dc4f" candidate_uuid: "8c59e1a935f64d6ca94e833af794dd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" is_pre_election: true
I20260927 00:52:54.604908 4770 raft_consensus.cc:2454] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c59e1a935f64d6ca94e833af794dd0b in term 0.
I20260927 00:52:54.605100 4903 raft_consensus.cc:2454] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c59e1a935f64d6ca94e833af794dd0b in term 0.
I20260927 00:52:54.605475 4573 leader_election.cc:302] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c59e1a935f64d6ca94e833af794dd0b, bb7d10c269984e88803aa684a5dd83f0; no voters:
I20260927 00:52:54.605798 4972 raft_consensus.cc:2781] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:54.605952 4972 raft_consensus.cc:482] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:54.605998 4972 raft_consensus.cc:3037] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:52:54.617105 4950 tablet.cc:2369] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:52:54.621140 4817 tablet.cc:2369] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:52:54.647264 4684 tablet.cc:2369] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:52:54.669864 4972 raft_consensus.cc:504] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.670599 4972 leader_election.cc:288] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [CANDIDATE]: Term 1 election: Requested vote from peers 926d0f3bc1bd4b2083b5ddcbcbe99fb6 (127.0.98.3:46639), bb7d10c269984e88803aa684a5dd83f0 (127.0.98.2:45075)
I20260927 00:52:54.671310 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a00ba34d944819a013e70a9ce9dc4f" candidate_uuid: "8c59e1a935f64d6ca94e833af794dd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6"
I20260927 00:52:54.671309 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a00ba34d944819a013e70a9ce9dc4f" candidate_uuid: "8c59e1a935f64d6ca94e833af794dd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7d10c269984e88803aa684a5dd83f0"
I20260927 00:52:54.671505 4903 raft_consensus.cc:3037] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:54.671505 4770 raft_consensus.cc:3037] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:54.673132 4770 raft_consensus.cc:2454] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c59e1a935f64d6ca94e833af794dd0b in term 1.
I20260927 00:52:54.673650 4573 leader_election.cc:302] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c59e1a935f64d6ca94e833af794dd0b, bb7d10c269984e88803aa684a5dd83f0; no voters:
I20260927 00:52:54.673933 4972 raft_consensus.cc:2781] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:54.674196 4972 raft_consensus.cc:686] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 1 LEADER]: Becoming Leader. State: Replica: 8c59e1a935f64d6ca94e833af794dd0b, State: Running, Role: LEADER
I20260927 00:52:54.674624 4972 consensus_queue.cc:237] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } }
I20260927 00:52:54.678610 4493 catalog_manager.cc:5495] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c59e1a935f64d6ca94e833af794dd0b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c59e1a935f64d6ca94e833af794dd0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c59e1a935f64d6ca94e833af794dd0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "926d0f3bc1bd4b2083b5ddcbcbe99fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 } health_report { overall_health: UNKNOWN } } }
I20260927 00:52:54.727475 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.723191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.0.98.1:41123)
0927 00:52:54.723403 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:54.723407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:54.723421 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:52:54.723532 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:54.723535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:54.723613 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:54.723723 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:52:54.724026 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.724747 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.725740 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:54.725975 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:54.726072 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:54.726489 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:52:54.727039 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:52:54.727192 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:54.727255 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:52:54.727325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 00:52:54.735309 4770 raft_consensus.cc:1261] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 1 FOLLOWER]: Refusing update from remote peer 8c59e1a935f64d6ca94e833af794dd0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:54.736348 4977 consensus_queue.cc:1035] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7d10c269984e88803aa684a5dd83f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:54.753963 4493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:54.754209 4493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:54.757164 4493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09bf9b17f83b4b09b39b1fd2ae581bff: Sending DeleteTablet for 3 replicas of tablet 00a00ba34d944819a013e70a9ce9dc4f
I20260927 00:52:54.758080 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a00ba34d944819a013e70a9ce9dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45298
I20260927 00:52:54.758080 4883 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a00ba34d944819a013e70a9ce9dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39168
I20260927 00:52:54.758633 4617 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a00ba34d944819a013e70a9ce9dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47328
I20260927 00:52:54.759078 4988 tablet_replica.cc:330] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b: stopping tablet replica
I20260927 00:52:54.759075 4986 tablet_replica.cc:330] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: stopping tablet replica
I20260927 00:52:54.759434 4988 raft_consensus.cc:2229] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:54.759434 4986 raft_consensus.cc:2229] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:54.759727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4555
I20260927 00:52:54.759912 4988 raft_consensus.cc:2258] T 00a00ba34d944819a013e70a9ce9dc4f P 8c59e1a935f64d6ca94e833af794dd0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:54.759945 4986 raft_consensus.cc:2258] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:54.760272 4987 tablet_replica.cc:330] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: stopping tablet replica
I20260927 00:52:54.762305 4986 ts_tablet_manager.cc:1905] T 00a00ba34d944819a013e70a9ce9dc4f P bb7d10c269984e88803aa684a5dd83f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:54.766614 4903 raft_consensus.cc:2454] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c59e1a935f64d6ca94e833af794dd0b in term 1.
W20260927 00:52:54.767107 4836 connection.cc:441] server connection from 127.0.98.1:42689 torn down before Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.98.1:42689 (request call id 1) could send its response
I20260927 00:52:54.767258 4902 raft_consensus.cc:1261] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 1 FOLLOWER]: Refusing update from remote peer 8c59e1a935f64d6ca94e833af794dd0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 00:52:54.767341 4477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41123 (error 108)
I20260927 00:52:54.767408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4688
W20260927 00:52:54.768177 4836 connection.cc:441] server connection from 127.0.98.1:42689 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.98.1:42689 (request call id 2) could send its response
I20260927 00:52:54.768467 4987 raft_consensus.cc:2229] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:54.768864 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.768475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41123 (local address 127.0.0.1:47348)
0927 00:52:54.768666 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.768764 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:52:54.768934 4987 raft_consensus.cc:2258] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:52:54.769109 4477 catalog_manager.cc:4584] TS 8c59e1a935f64d6ca94e833af794dd0b (127.0.98.1:41123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00a00ba34d944819a013e70a9ce9dc4f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41123: connect: Connection refused (error 111)
I20260927 00:52:54.770130 4987 ts_tablet_manager.cc:1905] T 00a00ba34d944819a013e70a9ce9dc4f P 926d0f3bc1bd4b2083b5ddcbcbe99fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:54.774492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4821
I20260927 00:52:54.774873 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:54.774570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45075 (local address 127.0.0.1:45306)
0927 00:52:54.774752 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:54.774796 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 00:52:54.790570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4462
2026-09-27T00:52:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2478 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T00:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:54Z Disabled control of system clock
I20260927 00:52:54.824084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33773
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:52:54.952916 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:54.953279 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:54.953320 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:54.957726 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:54.957827 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:54.957842 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:54.957859 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:54.957875 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:54.963128 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:54.964598 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:54.966284 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:54.976315 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:54.976440 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:54.976356 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:54.976758 4993 server_base.cc:1034] running on GCE node
I20260927 00:52:54.977484 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:54.978920 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:54.980119 4993 hybrid_clock.cc:648] HybridClock initialized: now 1790470374980081 us; error 60 us; skew 500 ppm
I20260927 00:52:54.988106 4993 webserver.cc:460] Webserver started at http://127.0.98.62:42977/ using document root <none> and password file <none>
I20260927 00:52:54.988911 4993 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:54.988992 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:54.989235 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:54.991997 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "8c6f78ef91d843248dc135917a54fc04"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "845b73291c6a917f98f254b46f471443"
server_key_iv: "391d591ffb0428d791110f2f8c70afac"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.992671 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c6f78ef91d843248dc135917a54fc04"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "845b73291c6a917f98f254b46f471443"
server_key_iv: "391d591ffb0428d791110f2f8c70afac"
server_key_version: "encryptionkey@0"
I20260927 00:52:54.997321 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:52:55.000830 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.002619 4993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:55.002780 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "8c6f78ef91d843248dc135917a54fc04"
format_stamp: "Formatted at 2026-09-27 00:52:54 on dist-test-slave-9fh2"
server_key: "845b73291c6a917f98f254b46f471443"
server_key_iv: "391d591ffb0428d791110f2f8c70afac"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.002923 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:55.008756 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:55.009784 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:55.010022 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:55.019397 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33773
I20260927 00:52:55.019434 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33773 every 8 connection(s)
I20260927 00:52:55.020658 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:55.021288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 4993
I20260927 00:52:55.021431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:55.021708 392 external_mini_cluster.cc:1428] Setting key ae7159033640bb55b2d87e9e456d3e69
I20260927 00:52:55.024597 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:55.029887 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.023646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.62:33773)
0927 00:52:55.024990 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.025014 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.025070 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:52:55.026099 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.026126 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.026669 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.027129 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.028007 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.028014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.028030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.028042 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.028119 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.028143 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.028314 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.028487 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.028803 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":604,"threads_started":1}
I20260927 00:52:55.032737 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: Bootstrap starting.
I20260927 00:52:55.035415 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:55.036374 5060 log.cc:824] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:55.039115 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: No bootstrap required, opened a new log
I20260927 00:52:55.043617 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f78ef91d843248dc135917a54fc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:52:55.043955 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:55.043995 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6f78ef91d843248dc135917a54fc04, State: Initialized, Role: FOLLOWER
I20260927 00:52:55.044553 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f78ef91d843248dc135917a54fc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:52:55.044701 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:55.044777 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:55.044891 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:55.046011 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f78ef91d843248dc135917a54fc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:52:55.046428 5060 leader_election.cc:302] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c6f78ef91d843248dc135917a54fc04; no voters:
I20260927 00:52:55.046797 5060 leader_election.cc:288] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:55.047297 5066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:55.047601 5066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [term 1 LEADER]: Becoming Leader. State: Replica: 8c6f78ef91d843248dc135917a54fc04, State: Running, Role: LEADER
I20260927 00:52:55.047971 5066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f78ef91d843248dc135917a54fc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } }
I20260927 00:52:55.048532 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:55.049943 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c6f78ef91d843248dc135917a54fc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f78ef91d843248dc135917a54fc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } } }
I20260927 00:52:55.050168 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:55.050575 5067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c6f78ef91d843248dc135917a54fc04. Latest consensus state: current_term: 1 leader_uuid: "8c6f78ef91d843248dc135917a54fc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f78ef91d843248dc135917a54fc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33773 } } }
I20260927 00:52:55.050681 5067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04 [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:55.054039 5080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:55.054136 5080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:55.054255 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:55.055194 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:55.061682 5077 catalog_manager.cc:1347] Generated new cluster ID: 6e32c7b4c15d478eb87e1f99c5f52efb
I20260927 00:52:55.061786 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:55.079862 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:55.080257 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:55.087724 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: Generated new TSK 0
I20260927 00:52:55.088820 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:52:55.106289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33773
--builtin_ntp_servers=127.0.98.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:52:55.256273 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:55.256666 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:55.256713 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:55.261620 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:55.261739 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:55.261833 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:55.267745 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:55.269291 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:55.271057 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:55.283087 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:55.283098 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:55.283605 5084 server_base.cc:1034] running on GCE node
W20260927 00:52:55.283098 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:55.284114 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:55.284900 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:55.286134 5084 hybrid_clock.cc:648] HybridClock initialized: now 1790470375286102 us; error 50 us; skew 500 ppm
I20260927 00:52:55.288372 5084 webserver.cc:460] Webserver started at http://127.0.98.1:44979/ using document root <none> and password file <none>
I20260927 00:52:55.289033 5084 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:55.289109 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:55.289320 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:55.291709 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2a7116a3e5246028f45c5a9a648ff6a"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "5b74a621c73788aeedad9ea53cfece78"
server_key_iv: "476aded108c6d3e0f945f645de77ca7e"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.292294 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a7116a3e5246028f45c5a9a648ff6a"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "5b74a621c73788aeedad9ea53cfece78"
server_key_iv: "476aded108c6d3e0f945f645de77ca7e"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.296690 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:55.299762 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.301402 5084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:55.301543 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "e2a7116a3e5246028f45c5a9a648ff6a"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "5b74a621c73788aeedad9ea53cfece78"
server_key_iv: "476aded108c6d3e0f945f645de77ca7e"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.301661 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:55.306684 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:55.307425 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:55.307601 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:55.308202 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:55.309185 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:55.309234 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.309286 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:55.309314 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.318784 5084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41253
I20260927 00:52:55.318815 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41253 every 8 connection(s)
I20260927 00:52:55.319818 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:55.324256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5084
I20260927 00:52:55.324386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:55.324681 392 external_mini_cluster.cc:1428] Setting key 715e8c0bed1da284c787b48f16d4e452
I20260927 00:52:55.327265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33773
--builtin_ntp_servers=127.0.98.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:52:55.327898 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.322293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45539 (local address 127.0.98.62:33773)
0927 00:52:55.322503 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.322507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.323898 (+ 1391us) server_negotiation.cc:407] Connection header received
0927 00:52:55.325551 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.325555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.325611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.325713 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.327320 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.327326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.327329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.327347 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.327378 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.327392 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.327501 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.327678 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.327732 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:52:55.328994 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.322766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33773 (local address 127.0.98.1:45539)
0927 00:52:55.323716 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.323766 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.325298 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.325952 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.325960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.326073 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.326914 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.326916 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.326967 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.327054 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.327506 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.327518 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.327538 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.327722 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.328034 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":166,"threads_started":1}
I20260927 00:52:55.330590 5212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33773
I20260927 00:52:55.330901 5212 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:55.331549 5212 heartbeater.cc:511] Master 127.0.98.62:33773 requested a full tablet report, sending...
I20260927 00:52:55.333433 5024 ts_manager.cc:194] Registered new tserver with Master: e2a7116a3e5246028f45c5a9a648ff6a (127.0.98.1:41253)
W20260927 00:52:55.458755 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:55.459101 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:55.459195 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:55.463550 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:55.463630 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:55.463719 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:55.468887 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:55.470479 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:55.472187 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:55.483453 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:55.483455 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:55.483603 5215 server_base.cc:1034] running on GCE node
W20260927 00:52:55.483831 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:55.484345 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:55.485126 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:55.486316 5215 hybrid_clock.cc:648] HybridClock initialized: now 1790470375486276 us; error 56 us; skew 500 ppm
I20260927 00:52:55.488421 5215 webserver.cc:460] Webserver started at http://127.0.98.2:34581/ using document root <none> and password file <none>
I20260927 00:52:55.489032 5215 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:55.489100 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:55.489339 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:55.491658 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "07160a6e59e44a68a5e910edaa906468"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "56e80e4236c426fb1b7f7d28f3e8caf5"
server_key_iv: "e24685e68a8e01f7b96e5bfce46cf1c1"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.492216 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "07160a6e59e44a68a5e910edaa906468"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "56e80e4236c426fb1b7f7d28f3e8caf5"
server_key_iv: "e24685e68a8e01f7b96e5bfce46cf1c1"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.496374 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:52:55.499214 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.500746 5215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:52:55.500885 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "07160a6e59e44a68a5e910edaa906468"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "56e80e4236c426fb1b7f7d28f3e8caf5"
server_key_iv: "e24685e68a8e01f7b96e5bfce46cf1c1"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.500999 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:55.505870 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:55.506520 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:55.506716 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:55.507329 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:55.508319 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:55.508368 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.508419 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:55.508447 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.518567 5215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36871
I20260927 00:52:55.518623 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36871 every 8 connection(s)
I20260927 00:52:55.519577 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:55.524811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5215
I20260927 00:52:55.524955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:55.525238 392 external_mini_cluster.cc:1428] Setting key 7cc224681cee0cd131555702d9c2e0df
I20260927 00:52:55.526911 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.521796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44319 (local address 127.0.98.62:33773)
0927 00:52:55.521961 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.521965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.523274 (+ 1309us) server_negotiation.cc:407] Connection header received
0927 00:52:55.524759 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.524763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.524823 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.524943 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.526456 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.526459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.526461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.526472 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.526492 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.526500 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.526570 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.526753 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.526800 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:52:55.527551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33773
--builtin_ntp_servers=127.0.98.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:52:55.528105 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.522233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33773 (local address 127.0.98.2:44319)
0927 00:52:55.523039 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.523094 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.524517 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.525197 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.525205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.525391 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.526079 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.526081 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.526142 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.526241 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.526582 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.526589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.526595 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.526794 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.527088 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260927 00:52:55.529507 5343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33773
I20260927 00:52:55.529749 5343 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:55.530182 5343 heartbeater.cc:511] Master 127.0.98.62:33773 requested a full tablet report, sending...
I20260927 00:52:55.531189 5024 ts_manager.cc:194] Registered new tserver with Master: 07160a6e59e44a68a5e910edaa906468 (127.0.98.2:36871)
W20260927 00:52:55.659822 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:55.660189 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:55.660235 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:55.664798 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:55.664894 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:55.664963 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:55.670804 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:55.672364 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:55.674151 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:55.685932 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:55.685997 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:55.685997 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:55.686357 5346 server_base.cc:1034] running on GCE node
I20260927 00:52:55.686906 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:55.687795 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:55.689014 5346 hybrid_clock.cc:648] HybridClock initialized: now 1790470375688966 us; error 63 us; skew 500 ppm
I20260927 00:52:55.691478 5346 webserver.cc:460] Webserver started at http://127.0.98.3:41651/ using document root <none> and password file <none>
I20260927 00:52:55.692215 5346 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:55.692292 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:55.692533 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:55.695093 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "25a247ed478c4ce1b9761b97bec2fab0"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "cb1021fcd8fb123d4bb47f808b9e88a7"
server_key_iv: "9fa5548acd3780d01b7806160e126e21"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.695767 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "25a247ed478c4ce1b9761b97bec2fab0"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "cb1021fcd8fb123d4bb47f808b9e88a7"
server_key_iv: "9fa5548acd3780d01b7806160e126e21"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.700412 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:55.703682 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.706159 5346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:55.706315 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "25a247ed478c4ce1b9761b97bec2fab0"
format_stamp: "Formatted at 2026-09-27 00:52:55 on dist-test-slave-9fh2"
server_key: "cb1021fcd8fb123d4bb47f808b9e88a7"
server_key_iv: "9fa5548acd3780d01b7806160e126e21"
server_key_version: "encryptionkey@0"
I20260927 00:52:55.706466 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:55.713536 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:55.714459 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:55.714659 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:55.715481 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:55.716800 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:55.716864 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.716923 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:55.716943 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:55.728037 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39641
I20260927 00:52:55.728163 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39641 every 8 connection(s)
I20260927 00:52:55.729146 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:55.735482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5346
I20260927 00:52:55.735600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:55.735942 392 external_mini_cluster.cc:1428] Setting key e13a0bd6f2d13817619e55aaa1b4a28d
I20260927 00:52:55.736533 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.731282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37471 (local address 127.0.98.62:33773)
0927 00:52:55.731490 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.731494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.732738 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 00:52:55.734218 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.734221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.734274 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.734353 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.735809 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.735814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.735817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.735831 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.735859 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.735870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.735980 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.736317 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.736385 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:52:55.737669 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.731683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33773 (local address 127.0.98.3:37471)
0927 00:52:55.732540 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.732599 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.733976 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.734567 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.734575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.734681 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.735443 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.735445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.735499 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.735619 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.735978 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.735985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.735990 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.736340 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.736635 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":136,"threads_started":1}
I20260927 00:52:55.738965 5474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33773
I20260927 00:52:55.739230 5474 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:55.739691 5474 heartbeater.cc:511] Master 127.0.98.62:33773 requested a full tablet report, sending...
I20260927 00:52:55.740774 5024 ts_manager.cc:194] Registered new tserver with Master: 25a247ed478c4ce1b9761b97bec2fab0 (127.0.98.3:39641)
I20260927 00:52:55.751135 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:52:55.755059 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.753544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36096 (local address 127.0.98.62:33773)
0927 00:52:55.753685 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.753688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.753833 (+ 145us) server_negotiation.cc:407] Connection header received
0927 00:52:55.753951 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.753954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.753996 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.754083 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.754722 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.754725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.754727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.754737 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.754758 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.754767 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.754836 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.754896 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.754933 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:52:55.758394 5024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:52:55.762074 5024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:52:55.775255 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.771750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36871 (local address 127.0.0.1:54386)
0927 00:52:55.772199 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.772245 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.772348 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.773183 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.773188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.773264 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.773915 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.773918 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.773942 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.774023 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.774842 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.774851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.774857 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.774980 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.775055 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":84,"threads_started":1}
I20260927 00:52:55.775471 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.772466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39641 (local address 127.0.0.1:43848)
0927 00:52:55.772976 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.772991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.773079 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.773970 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.773973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.773983 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.774405 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.774407 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.774420 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.774453 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.775269 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.775272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.775276 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.775330 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.775364 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":112,"threads_started":1}
I20260927 00:52:55.775848 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.772789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41253 (local address 127.0.0.1:55138)
0927 00:52:55.773299 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.773312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.773394 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.773923 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.773926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.773943 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.774504 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.774506 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.774515 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.774537 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.775632 (+ 1095us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.775635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.775639 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.775710 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.775742 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":74,"threads_started":1}
I20260927 00:52:55.775873 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.771905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.0.98.2:36871)
0927 00:52:55.772592 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.772607 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.772636 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:52:55.772703 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.772710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.773001 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.773624 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.774640 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.774647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.774661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.774670 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.774714 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.774738 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.774863 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.775163 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.775258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":165,"threads_started":1}
I20260927 00:52:55.776078 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.772660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.0.98.3:39641)
0927 00:52:55.773548 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.773565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.773591 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:52:55.773644 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.773649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.773835 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.774185 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.775045 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.775051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.775073 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.775083 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.775121 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.775172 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.775690 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.775738 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.775808 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":143,"threads_started":1}
I20260927 00:52:55.776297 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.773166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.98.1:41253)
0927 00:52:55.773523 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.773536 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.773561 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:52:55.773617 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.773622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.773776 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.773909 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.774640 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.774647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.774661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.774670 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.774714 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.774738 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.774998 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.776046 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.776129 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260927 00:52:55.778894 5408 tablet_service.cc:1467] Processing CreateTablet for tablet f34f66041d244b97b0cedcbea73f41d1 (DEFAULT_TABLE table=test-table [id=c5393dfe23b440c2b9455daa42368e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:55.779232 5146 tablet_service.cc:1467] Processing CreateTablet for tablet f34f66041d244b97b0cedcbea73f41d1 (DEFAULT_TABLE table=test-table [id=c5393dfe23b440c2b9455daa42368e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:55.779232 5272 tablet_service.cc:1467] Processing CreateTablet for tablet f34f66041d244b97b0cedcbea73f41d1 (DEFAULT_TABLE table=test-table [id=c5393dfe23b440c2b9455daa42368e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:52:55.780007 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f34f66041d244b97b0cedcbea73f41d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:55.780390 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f34f66041d244b97b0cedcbea73f41d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:55.780390 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f34f66041d244b97b0cedcbea73f41d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:55.786176 5491 tablet_bootstrap.cc:492] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Bootstrap starting.
I20260927 00:52:55.788609 5493 tablet_bootstrap.cc:492] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Bootstrap starting.
I20260927 00:52:55.789292 5491 tablet_bootstrap.cc:654] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:55.789831 5492 tablet_bootstrap.cc:492] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: Bootstrap starting.
I20260927 00:52:55.790182 5491 log.cc:824] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:55.790762 5493 tablet_bootstrap.cc:654] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:55.791482 5493 log.cc:824] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:55.792248 5492 tablet_bootstrap.cc:654] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:55.792580 5491 tablet_bootstrap.cc:492] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: No bootstrap required, opened a new log
I20260927 00:52:55.792778 5491 ts_tablet_manager.cc:1397] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:52:55.793146 5492 log.cc:824] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:55.793462 5493 tablet_bootstrap.cc:492] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: No bootstrap required, opened a new log
I20260927 00:52:55.793637 5493 ts_tablet_manager.cc:1397] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:52:55.795418 5492 tablet_bootstrap.cc:492] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: No bootstrap required, opened a new log
I20260927 00:52:55.795617 5492 ts_tablet_manager.cc:1397] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:52:55.797843 5493 raft_consensus.cc:348] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.797823 5491 raft_consensus.cc:348] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.798121 5493 raft_consensus.cc:374] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:55.798121 5491 raft_consensus.cc:374] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:55.798170 5491 raft_consensus.cc:729] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a247ed478c4ce1b9761b97bec2fab0, State: Initialized, Role: FOLLOWER
I20260927 00:52:55.798170 5493 raft_consensus.cc:729] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07160a6e59e44a68a5e910edaa906468, State: Initialized, Role: FOLLOWER
I20260927 00:52:55.798689 5491 consensus_queue.cc:260] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.798689 5493 consensus_queue.cc:260] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.799516 5474 heartbeater.cc:503] Master 127.0.98.62:33773 was elected leader, sending a full tablet report...
I20260927 00:52:55.799791 5493 ts_tablet_manager.cc:1428] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:52:55.799906 5343 heartbeater.cc:503] Master 127.0.98.62:33773 was elected leader, sending a full tablet report...
I20260927 00:52:55.799783 5492 raft_consensus.cc:348] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.800374 5492 raft_consensus.cc:374] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:55.800433 5492 raft_consensus.cc:729] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7116a3e5246028f45c5a9a648ff6a, State: Initialized, Role: FOLLOWER
I20260927 00:52:55.800967 5492 consensus_queue.cc:260] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.801615 5212 heartbeater.cc:503] Master 127.0.98.62:33773 was elected leader, sending a full tablet report...
I20260927 00:52:55.802048 5492 ts_tablet_manager.cc:1428] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:52:55.803046 5491 ts_tablet_manager.cc:1428] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Time spent starting tablet: real 0.010s user 0.000s sys 0.007s
W20260927 00:52:55.821260 5213 tablet.cc:2369] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:52:55.841202 5497 raft_consensus.cc:482] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:52:55.841436 5497 raft_consensus.cc:504] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.842614 5497 leader_election.cc:288] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25a247ed478c4ce1b9761b97bec2fab0 (127.0.98.3:39641), e2a7116a3e5246028f45c5a9a648ff6a (127.0.98.1:41253)
I20260927 00:52:55.844900 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.843204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41399 (local address 127.0.98.3:39641)
0927 00:52:55.843335 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.843339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.843356 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:52:55.843460 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.843464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.843534 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.843619 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.844489 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.844499 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.844502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.844511 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.844535 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.844545 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.844639 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.844746 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.844787 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:52:55.844923 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.843000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39641 (local address 127.0.98.2:41399)
0927 00:52:55.843198 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.843217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.843327 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.843673 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.843676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.843710 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.844329 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.844330 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.844345 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.844378 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.844652 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.844655 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.844658 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.844725 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.844766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:52:55.845330 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.843440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41253 (local address 127.0.98.2:50713)
0927 00:52:55.843901 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.843916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:55.844008 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:55.844221 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:55.844224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:55.844238 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:55.844636 (+ 398us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:52:55.844638 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:55.844651 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:52:55.844682 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:55.845041 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:55.845045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:55.845047 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 00:52:55.845157 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:52:55.845194 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":148,"threads_started":1}
I20260927 00:52:55.845367 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.843620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50713 (local address 127.0.98.1:41253)
0927 00:52:55.843786 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.843790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.844016 (+ 226us) server_negotiation.cc:407] Connection header received
0927 00:52:55.844083 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.844086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.844127 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.844194 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.844849 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.844853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.844856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.844869 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.844895 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.844906 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.844999 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.845183 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.845231 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:52:55.845818 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34f66041d244b97b0cedcbea73f41d1" candidate_uuid: "07160a6e59e44a68a5e910edaa906468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a247ed478c4ce1b9761b97bec2fab0" is_pre_election: true
I20260927 00:52:55.845876 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34f66041d244b97b0cedcbea73f41d1" candidate_uuid: "07160a6e59e44a68a5e910edaa906468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" is_pre_election: true
I20260927 00:52:55.846143 5428 raft_consensus.cc:2454] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07160a6e59e44a68a5e910edaa906468 in term 0.
I20260927 00:52:55.846143 5166 raft_consensus.cc:2454] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07160a6e59e44a68a5e910edaa906468 in term 0.
I20260927 00:52:55.846649 5233 leader_election.cc:302] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07160a6e59e44a68a5e910edaa906468, e2a7116a3e5246028f45c5a9a648ff6a; no voters:
I20260927 00:52:55.846908 5497 raft_consensus.cc:2781] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:52:55.847011 5497 raft_consensus.cc:482] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:52:55.847043 5497 raft_consensus.cc:3037] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:55.848346 5497 raft_consensus.cc:504] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.848798 5497 leader_election.cc:288] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [CANDIDATE]: Term 1 election: Requested vote from peers 25a247ed478c4ce1b9761b97bec2fab0 (127.0.98.3:39641), e2a7116a3e5246028f45c5a9a648ff6a (127.0.98.1:41253)
I20260927 00:52:55.849283 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34f66041d244b97b0cedcbea73f41d1" candidate_uuid: "07160a6e59e44a68a5e910edaa906468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7116a3e5246028f45c5a9a648ff6a"
I20260927 00:52:55.849293 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34f66041d244b97b0cedcbea73f41d1" candidate_uuid: "07160a6e59e44a68a5e910edaa906468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a247ed478c4ce1b9761b97bec2fab0"
I20260927 00:52:55.849438 5428 raft_consensus.cc:3037] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:55.849449 5166 raft_consensus.cc:3037] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:55.850636 5428 raft_consensus.cc:2454] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07160a6e59e44a68a5e910edaa906468 in term 1.
I20260927 00:52:55.850699 5166 raft_consensus.cc:2454] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07160a6e59e44a68a5e910edaa906468 in term 1.
I20260927 00:52:55.850961 5233 leader_election.cc:302] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07160a6e59e44a68a5e910edaa906468, 25a247ed478c4ce1b9761b97bec2fab0; no voters:
I20260927 00:52:55.851166 5497 raft_consensus.cc:2781] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:55.851435 5497 raft_consensus.cc:686] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 1 LEADER]: Becoming Leader. State: Replica: 07160a6e59e44a68a5e910edaa906468, State: Running, Role: LEADER
I20260927 00:52:55.851881 5497 consensus_queue.cc:237] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } }
I20260927 00:52:55.855759 5023 catalog_manager.cc:5495] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07160a6e59e44a68a5e910edaa906468 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "07160a6e59e44a68a5e910edaa906468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07160a6e59e44a68a5e910edaa906468" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 } health_report { overall_health: UNKNOWN } } }
I20260927 00:52:55.878288 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.876626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.0.98.2:36871)
0927 00:52:55.876792 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:55.876814 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:55.876831 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:52:55.876932 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:55.876935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:55.877022 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:55.877105 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:55.877878 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:55.877881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:55.877883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:55.877894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:55.877915 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:55.877923 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:55.878012 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:55.878080 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:52:55.878118 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:52:55.885581 5166 raft_consensus.cc:1261] T f34f66041d244b97b0cedcbea73f41d1 P e2a7116a3e5246028f45c5a9a648ff6a [term 1 FOLLOWER]: Refusing update from remote peer 07160a6e59e44a68a5e910edaa906468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:55.885597 5428 raft_consensus.cc:1261] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 1 FOLLOWER]: Refusing update from remote peer 07160a6e59e44a68a5e910edaa906468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:52:55.886358 5497 consensus_queue.cc:1035] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25a247ed478c4ce1b9761b97bec2fab0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:55.886595 5501 consensus_queue.cc:1035] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a7116a3e5246028f45c5a9a648ff6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:52:55.897408 5506 mvcc.cc:205] Tried to move back new op lower bound from 7333766659618422784 to 7333766659492417536. Current Snapshot: MvccSnapshot[applied={T|T < 7333766659618422784}]
I20260927 00:52:55.905112 5024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:55.905370 5024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:52:55.908646 5024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c6f78ef91d843248dc135917a54fc04: Sending DeleteTablet for 3 replicas of tablet f34f66041d244b97b0cedcbea73f41d1
I20260927 00:52:55.909607 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet f34f66041d244b97b0cedcbea73f41d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:55 UTC) from {username='slave'} at 127.0.0.1:54386
I20260927 00:52:55.909781 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet f34f66041d244b97b0cedcbea73f41d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:52:55 UTC) from {username='slave'} at 127.0.0.1:43848
I20260927 00:52:55.910317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5084
I20260927 00:52:55.912432 5513 tablet_replica.cc:330] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: stopping tablet replica
I20260927 00:52:55.912878 5513 raft_consensus.cc:2229] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:52:55.913349 5513 raft_consensus.cc:2258] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:55.914983 5513 ts_tablet_manager.cc:1905] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:55.915412 5514 tablet_replica.cc:330] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: stopping tablet replica
I20260927 00:52:55.915889 5514 raft_consensus.cc:2229] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:52:55.916366 5514 raft_consensus.cc:2258] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:52:55.918064 5513 ts_tablet_manager.cc:1918] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:52:55.918170 5513 log.cc:1196] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/f34f66041d244b97b0cedcbea73f41d1
W20260927 00:52:55.918231 5011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41253 (error 108)
I20260927 00:52:55.918495 5513 ts_tablet_manager.cc:1939] T f34f66041d244b97b0cedcbea73f41d1 P 07160a6e59e44a68a5e910edaa906468: Deleting consensus metadata
I20260927 00:52:55.919016 5514 ts_tablet_manager.cc:1905] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:52:55.919250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5215
I20260927 00:52:55.921528 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.921090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41253 (local address 127.0.0.1:55154)
0927 00:52:55.921329 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.921414 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260927 00:52:55.921758 5011 catalog_manager.cc:4584] TS e2a7116a3e5246028f45c5a9a648ff6a (127.0.98.1:41253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f34f66041d244b97b0cedcbea73f41d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41253: connect: Connection refused (error 111)
I20260927 00:52:55.923228 5514 ts_tablet_manager.cc:1918] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:52:55.923337 5514 log.cc:1196] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/f34f66041d244b97b0cedcbea73f41d1
I20260927 00:52:55.923615 5514 ts_tablet_manager.cc:1939] T f34f66041d244b97b0cedcbea73f41d1 P 25a247ed478c4ce1b9761b97bec2fab0: Deleting consensus metadata
I20260927 00:52:55.925150 5011 catalog_manager.cc:4841] TS 25a247ed478c4ce1b9761b97bec2fab0 (127.0.98.3:39641): tablet f34f66041d244b97b0cedcbea73f41d1 (table test-table [id=c5393dfe23b440c2b9455daa42368e51]) successfully deleted
I20260927 00:52:55.925755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5346
I20260927 00:52:55.927857 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:55.927521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36871 (local address 127.0.0.1:54400)
0927 00:52:55.927682 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:55.927750 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:52:55.931293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 4993
2026-09-27T00:52:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1137 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 00:52:55.951985 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:52:55.953600 392 webserver.cc:460] Webserver started at https://127.0.0.1:40107/ using document root <none> and password file <none>
I20260927 00:52:55.953768 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:52:55.954035 392 webserver.cc:460] Webserver started at http://127.0.0.1:33781/ using document root <none> and password file <none>
2026-09-27T00:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:55Z Disabled control of system clock
I20260927 00:52:55.963944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36297
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40107/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/testchainca.cert with env {}
W20260927 00:52:56.094205 5521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:56.094573 5521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:56.094635 5521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:56.099097 5521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:56.099211 5521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:56.099236 5521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:56.099265 5521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:56.099287 5521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:52:56.099334 5521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:52:56.099361 5521 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40107/jwks.json
I20260927 00:52:56.105659 5521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40107/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:56.107100 5521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:56.108412 5521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:56.118327 5529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:56.118395 5527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:56.118454 5521 server_base.cc:1034] running on GCE node
W20260927 00:52:56.118327 5526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:56.119426 5521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:56.120704 5521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:56.121892 5521 hybrid_clock.cc:648] HybridClock initialized: now 1790470376121863 us; error 49 us; skew 500 ppm
I20260927 00:52:56.124032 5521 webserver.cc:460] Webserver started at http://127.0.98.62:41103/ using document root <none> and password file <none>
I20260927 00:52:56.124683 5521 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:56.124754 5521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:56.124958 5521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:56.127259 5521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "9b83a5f8308a4cc29ba9fa4b191578e9"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "52806014ef7395fbbec9e9e3c6f0f15c"
server_key_iv: "4735018bb478b9d419f9cb5642b90d87"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.127839 5521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b83a5f8308a4cc29ba9fa4b191578e9"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "52806014ef7395fbbec9e9e3c6f0f15c"
server_key_iv: "4735018bb478b9d419f9cb5642b90d87"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.132046 5521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:52:56.134968 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:56.136730 5521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:52:56.136879 5521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "9b83a5f8308a4cc29ba9fa4b191578e9"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "52806014ef7395fbbec9e9e3c6f0f15c"
server_key_iv: "4735018bb478b9d419f9cb5642b90d87"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.137012 5521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:56.167928 5521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:56.168684 5521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:56.168866 5521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:56.176110 5521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36297
I20260927 00:52:56.176203 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36297 every 8 connection(s)
I20260927 00:52:56.177228 5521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:56.181449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5521
I20260927 00:52:56.181607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:56.181612 5588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:56.181969 392 external_mini_cluster.cc:1428] Setting key 78aa4a3ec559bfd194e3c3c9ecdadb76
I20260927 00:52:56.190608 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9: Bootstrap starting.
I20260927 00:52:56.193399 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:56.183785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.0.98.62:36297)
0927 00:52:56.184500 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:56.184517 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:56.184558 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:52:56.185449 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:56.185469 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:56.185926 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:56.186272 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:56.186796 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.187929 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.189363 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.189931 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.190089 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.191236 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:56.191243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:56.191260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:56.191735 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:56.191806 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:56.191828 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:56.191921 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:56.192133 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:52:56.192507 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":134,"threads_started":1}
I20260927 00:52:56.193981 5588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:56.194969 5588 log.cc:824] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:56.197358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:52:56.197836 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9: No bootstrap required, opened a new log
I20260927 00:52:56.202180 5588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } }
I20260927 00:52:56.202507 5588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:56.202560 5588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b83a5f8308a4cc29ba9fa4b191578e9, State: Initialized, Role: FOLLOWER
I20260927 00:52:56.203081 5588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } }
I20260927 00:52:56.203285 5588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:56.203356 5588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:56.203472 5588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:56.204813 5588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } }
I20260927 00:52:56.205232 5588 leader_election.cc:302] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b83a5f8308a4cc29ba9fa4b191578e9; no voters:
I20260927 00:52:56.205621 5588 leader_election.cc:288] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:56.205798 5593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:56.206194 5593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [term 1 LEADER]: Becoming Leader. State: Replica: 9b83a5f8308a4cc29ba9fa4b191578e9, State: Running, Role: LEADER
I20260927 00:52:56.206604 5593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } }
I20260927 00:52:56.207103 5588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:56.208448 5595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b83a5f8308a4cc29ba9fa4b191578e9. Latest consensus state: current_term: 1 leader_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } }
I20260927 00:52:56.208621 5595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:56.208966 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83a5f8308a4cc29ba9fa4b191578e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } }
I20260927 00:52:56.209055 5602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:56.209080 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:56.212457 5602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:56.218284 5602 catalog_manager.cc:1347] Generated new cluster ID: 9bbd7d3589414b4b94c92c27452a7252
I20260927 00:52:56.218385 5602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:56.235123 5602 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:56.236133 5602 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:56.241231 5602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b83a5f8308a4cc29ba9fa4b191578e9: Generated new TSK 0
I20260927 00:52:56.242121 5602 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:52:56.278891 5618 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 00:52:56.279021 5618 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:52:56.279616 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:56.262794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.98.62:36297)
0927 00:52:56.262948 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:56.262953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:56.263173 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:52:56.263403 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:56.263406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:56.263462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:56.263546 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:52:56.263785 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.264356 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.265210 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.265442 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.265526 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.265874 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:52:56.279194 (+ 13320us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:52:56.279474 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:52:56.279712 5591 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 00:52:56.280808 5617 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:56.262596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36297 (local address 127.0.0.1:33366)
0927 00:52:56.263045 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:56.263067 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:56.263231 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:56.263568 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:56.263571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:56.263579 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:52:56.263690 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:56.263696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.264464 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.264468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:56.265103 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:56.265111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.265526 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.265550 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:56.265636 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.265681 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:52:56.279425 (+ 13744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:52:56.279669 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:36297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 00:52:56.281435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5521
2026-09-27T00:52:56Z chronyd exiting
I20260927 00:52:56.319453 392 test_util.cc:175] -----------------------------------------------
I20260927 00:52:56.319545 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470348013064-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (369 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 00:52:56.320540 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:52:56.321398 392 webserver.cc:460] Webserver started at https://127.0.0.1:44419/ using document root <none> and password file <none>
I20260927 00:52:56.321489 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:52:56.321712 392 webserver.cc:460] Webserver started at http://127.0.0.1:43633/ using document root <none> and password file <none>
2026-09-27T00:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:56Z Disabled control of system clock
I20260927 00:52:56.331030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44419/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/testchainca.cert with env {}
W20260927 00:52:56.448365 5624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:56.448671 5624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:56.448714 5624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:56.452791 5624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:56.452870 5624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:56.452885 5624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:56.452903 5624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:56.452919 5624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:52:56.452934 5624 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:52:56.452946 5624 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44419/jwks.json
I20260927 00:52:56.458523 5624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44419/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:56.459666 5624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:56.460698 5624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:56.469458 5632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:56.469547 5630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:56.469946 5624 server_base.cc:1034] running on GCE node
W20260927 00:52:56.470484 5629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:56.471072 5624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:56.472244 5624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:56.473426 5624 hybrid_clock.cc:648] HybridClock initialized: now 1790470376473403 us; error 42 us; skew 500 ppm
I20260927 00:52:56.480369 5624 webserver.cc:460] Webserver started at http://127.0.98.62:36303/ using document root <none> and password file <none>
I20260927 00:52:56.480985 5624 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:56.481055 5624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:56.481245 5624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:56.483295 5624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "16f996d06c3047838c5ad7a22528c869"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "a75d92d85d1662cc04054258f354166f"
server_key_iv: "00df5b8125bcbe24b3817a406415c15c"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.483839 5624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "16f996d06c3047838c5ad7a22528c869"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "a75d92d85d1662cc04054258f354166f"
server_key_iv: "00df5b8125bcbe24b3817a406415c15c"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.487738 5624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:56.490451 5638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:56.492094 5624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:56.492236 5624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "16f996d06c3047838c5ad7a22528c869"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "a75d92d85d1662cc04054258f354166f"
server_key_iv: "00df5b8125bcbe24b3817a406415c15c"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.492372 5624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:56.508119 5624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:56.508801 5624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:56.508977 5624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:56.516036 5624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37501
I20260927 00:52:56.516088 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37501 every 8 connection(s)
I20260927 00:52:56.517001 5624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:56.518847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5624
I20260927 00:52:56.518988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:56.519397 392 external_mini_cluster.cc:1428] Setting key 8d77b8f2773c48e62e2f6872d97e3c45
I20260927 00:52:56.521473 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:56.528323 5691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869: Bootstrap starting.
I20260927 00:52:56.529253 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:56.521098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.98.62:37501)
0927 00:52:56.521961 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:56.521981 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:56.522021 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:52:56.522691 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:56.522709 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:56.523192 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:56.523523 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:56.523953 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.524949 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.525768 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.526250 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.526395 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.527412 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:56.527418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:56.527431 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:56.527844 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:56.527904 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:56.527925 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:56.528014 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:56.528157 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 00:52:56.528420 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":188,"threads_started":1}
I20260927 00:52:56.531270 5691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:56.532243 5691 log.cc:824] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:56.532685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:52:56.535039 5691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869: No bootstrap required, opened a new log
I20260927 00:52:56.539453 5691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f996d06c3047838c5ad7a22528c869" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260927 00:52:56.539780 5691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:56.539832 5691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f996d06c3047838c5ad7a22528c869, State: Initialized, Role: FOLLOWER
I20260927 00:52:56.540335 5691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f996d06c3047838c5ad7a22528c869" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260927 00:52:56.540494 5691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:56.540565 5691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:56.540668 5691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:56.541838 5691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f996d06c3047838c5ad7a22528c869" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260927 00:52:56.542275 5691 leader_election.cc:302] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f996d06c3047838c5ad7a22528c869; no voters:
I20260927 00:52:56.542672 5691 leader_election.cc:288] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:56.542896 5696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:56.543303 5696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [term 1 LEADER]: Becoming Leader. State: Replica: 16f996d06c3047838c5ad7a22528c869, State: Running, Role: LEADER
I20260927 00:52:56.543740 5696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f996d06c3047838c5ad7a22528c869" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260927 00:52:56.544355 5691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:56.545593 5698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f996d06c3047838c5ad7a22528c869. Latest consensus state: current_term: 1 leader_uuid: "16f996d06c3047838c5ad7a22528c869" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f996d06c3047838c5ad7a22528c869" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260927 00:52:56.545754 5698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:56.546222 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16f996d06c3047838c5ad7a22528c869" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f996d06c3047838c5ad7a22528c869" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260927 00:52:56.546397 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869 [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:56.549602 5711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:56.549688 5711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:56.549784 5705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:56.550470 5705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:56.556057 5705 catalog_manager.cc:1347] Generated new cluster ID: 1acf1367fba246ea8a18f1a918eb7339
I20260927 00:52:56.556140 5705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:56.573562 5705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:56.574515 5705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:56.579942 5705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16f996d06c3047838c5ad7a22528c869: Generated new TSK 0
I20260927 00:52:56.580617 5705 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:52:56.618593 5721 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 00:52:56.618721 5721 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:52:56.619385 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:56.595475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.98.62:37501)
0927 00:52:56.595588 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:56.595592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:56.595920 (+ 328us) server_negotiation.cc:407] Connection header received
0927 00:52:56.596098 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:56.596100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:56.596149 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:56.596315 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:52:56.596580 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.597187 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.598076 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.598313 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.598409 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.598695 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:52:56.618905 (+ 20210us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:52:56.619227 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:52:56.619478 5694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 00:52:56.619537 5720 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:56.595460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.0.1:39838)
0927 00:52:56.595831 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:56.595845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:56.595958 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:56.596370 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:56.596373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:56.596382 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:52:56.596483 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:56.596488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.597299 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.597303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:56.597965 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:56.597974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.598397 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.598421 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:56.598506 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.598534 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:52:56.619130 (+ 20596us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:52:56.619358 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44419/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":153,"threads_started":1}
I20260927 00:52:56.620270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5624
2026-09-27T00:52:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 00:52:56.659952 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:52:56.660799 392 webserver.cc:460] Webserver started at https://127.0.0.1:35053/ using document root <none> and password file <none>
I20260927 00:52:56.660890 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:52:56.661113 392 webserver.cc:460] Webserver started at http://127.0.0.1:42547/ using document root <none> and password file <none>
2026-09-27T00:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:56Z Disabled control of system clock
I20260927 00:52:56.670799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34631
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34631
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35053/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:52:56.781167 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:56.781452 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:56.781490 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:56.785486 5727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:56.785573 5727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:56.785588 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:56.785604 5727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:56.785619 5727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:52:56.785635 5727 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:52:56.785650 5727 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35053/jwks.json
I20260927 00:52:56.790597 5727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35053/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:56.791826 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:56.793182 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:56.802192 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:56.802192 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:56.802196 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:56.802242 5727 server_base.cc:1034] running on GCE node
I20260927 00:52:56.803133 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:56.804303 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:56.805501 5727 hybrid_clock.cc:648] HybridClock initialized: now 1790470376805481 us; error 43 us; skew 500 ppm
I20260927 00:52:56.807562 5727 webserver.cc:460] Webserver started at http://127.0.98.62:44999/ using document root <none> and password file <none>
I20260927 00:52:56.808122 5727 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:56.808190 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:56.808377 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:56.810405 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "b0e2b45ad7c64b168d7b64066d292c63"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "8a9622f211d0a88783e5a5f0edcf2528"
server_key_iv: "d98f9a7733915ac452fd7ecb519272dd"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.810958 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0e2b45ad7c64b168d7b64066d292c63"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "8a9622f211d0a88783e5a5f0edcf2528"
server_key_iv: "d98f9a7733915ac452fd7ecb519272dd"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.814811 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:52:56.817516 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:56.818946 5727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:56.819078 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "b0e2b45ad7c64b168d7b64066d292c63"
format_stamp: "Formatted at 2026-09-27 00:52:56 on dist-test-slave-9fh2"
server_key: "8a9622f211d0a88783e5a5f0edcf2528"
server_key_iv: "d98f9a7733915ac452fd7ecb519272dd"
server_key_version: "encryptionkey@0"
I20260927 00:52:56.819227 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:56.843611 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:56.844317 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:56.844506 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:56.851670 5727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34631
I20260927 00:52:56.851711 5793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34631 every 8 connection(s)
I20260927 00:52:56.852648 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:56.856832 5794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:56.857225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5727
I20260927 00:52:56.857353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:56.857693 392 external_mini_cluster.cc:1428] Setting key a0bc08d83bfa82ada9cf8fdac7e50f02
I20260927 00:52:56.865842 5794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63: Bootstrap starting.
I20260927 00:52:56.868399 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:56.859421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.0.98.62:34631)
0927 00:52:56.860169 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:56.860186 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:56.860226 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:52:56.861059 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:56.861084 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:56.861518 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:56.861837 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:56.862366 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.863516 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.864786 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.865331 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.865470 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.866501 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:56.866506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:56.866521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:56.866936 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:56.866998 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:56.867018 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:56.867108 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:56.867312 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 00:52:56.867587 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":160,"threads_started":1}
I20260927 00:52:56.868953 5794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:56.869937 5794 log.cc:824] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:56.871961 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:52:56.872249 5794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63: No bootstrap required, opened a new log
I20260927 00:52:56.875953 5794 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e2b45ad7c64b168d7b64066d292c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34631 } }
I20260927 00:52:56.876200 5794 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:56.876246 5794 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e2b45ad7c64b168d7b64066d292c63, State: Initialized, Role: FOLLOWER
I20260927 00:52:56.876713 5794 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e2b45ad7c64b168d7b64066d292c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34631 } }
I20260927 00:52:56.876868 5794 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:56.876940 5794 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:56.877048 5794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:56.878027 5794 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e2b45ad7c64b168d7b64066d292c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34631 } }
I20260927 00:52:56.878360 5794 leader_election.cc:302] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e2b45ad7c64b168d7b64066d292c63; no voters:
I20260927 00:52:56.878657 5794 leader_election.cc:288] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:56.878865 5799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:56.879196 5799 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [term 1 LEADER]: Becoming Leader. State: Replica: b0e2b45ad7c64b168d7b64066d292c63, State: Running, Role: LEADER
I20260927 00:52:56.879634 5799 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e2b45ad7c64b168d7b64066d292c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34631 } }
I20260927 00:52:56.879899 5794 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:56.881855 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e2b45ad7c64b168d7b64066d292c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e2b45ad7c64b168d7b64066d292c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34631 } } }
I20260927 00:52:56.882010 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:56.882253 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e2b45ad7c64b168d7b64066d292c63. Latest consensus state: current_term: 1 leader_uuid: "b0e2b45ad7c64b168d7b64066d292c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e2b45ad7c64b168d7b64066d292c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34631 } } }
I20260927 00:52:56.882680 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63 [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:56.885478 5813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:56.885552 5813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:56.885641 5814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:56.886409 5814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:56.892504 5814 catalog_manager.cc:1347] Generated new cluster ID: f5474599dab0464ba2d383fd2ac4c34a
I20260927 00:52:56.892601 5814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:56.899705 5814 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:56.900930 5814 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:56.915369 5814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e2b45ad7c64b168d7b64066d292c63: Generated new TSK 0
I20260927 00:52:56.916399 5814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:52:56.932533 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:56.926390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.0.98.62:34631)
0927 00:52:56.926550 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:56.926556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:56.926751 (+ 195us) server_negotiation.cc:407] Connection header received
0927 00:52:56.926941 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:56.926945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:56.927016 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:56.927214 (+ 198us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:52:56.927461 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.928387 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.929301 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.929669 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.929774 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.930201 (+ 427us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:52:56.932065 (+ 1864us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:52:56.932349 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:52:56.932633 5797 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 00:52:56.934105 5823 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:56.926237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34631 (local address 127.0.0.1:48674)
0927 00:52:56.926661 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:56.926674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:56.926783 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:56.927216 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:56.927219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:56.927228 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:52:56.927347 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:56.927352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.928518 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.928521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:56.929195 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:56.929202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:56.929790 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:56.929817 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:56.929901 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:56.929930 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:52:56.932249 (+ 2319us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:52:56.933918 (+ 1669us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34631: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35053/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1394,"thread_start_us":147,"threads_started":1}
I20260927 00:52:56.935029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5727
E20260927 00:52:56.945269 5824 webserver.cc:533] Webserver: sslize failed:
E20260927 00:52:56.945362 5824 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T00:52:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T00:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:56Z Disabled control of system clock
I20260927 00:52:56.986486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41557
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41557
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/testchainca.cert with env {}
W20260927 00:52:57.115044 5828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:57.115420 5828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:57.115468 5828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:57.119752 5828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:57.119836 5828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:57.119850 5828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:57.119867 5828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:57.119882 5828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:52:57.119899 5828 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:52:57.119910 5828 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 00:52:57.125907 5828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:57.127283 5828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:57.128504 5828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:57.138581 5833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:57.138530 5834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:57.138716 5828 server_base.cc:1034] running on GCE node
W20260927 00:52:57.138986 5836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:57.139779 5828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:57.141144 5828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:57.142338 5828 hybrid_clock.cc:648] HybridClock initialized: now 1790470377142311 us; error 39 us; skew 500 ppm
I20260927 00:52:57.144771 5828 webserver.cc:460] Webserver started at http://127.0.98.62:33599/ using document root <none> and password file <none>
I20260927 00:52:57.145488 5828 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:57.145567 5828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:57.145803 5828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:57.148382 5828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "05d07d63053d41cb935b944f3dddc68e"
format_stamp: "Formatted at 2026-09-27 00:52:57 on dist-test-slave-9fh2"
server_key: "a7c2a36162063ebf2de8c7c399771979"
server_key_iv: "1f5a0accaf7f9c7c418fa67aff70170c"
server_key_version: "encryptionkey@0"
I20260927 00:52:57.149088 5828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "05d07d63053d41cb935b944f3dddc68e"
format_stamp: "Formatted at 2026-09-27 00:52:57 on dist-test-slave-9fh2"
server_key: "a7c2a36162063ebf2de8c7c399771979"
server_key_iv: "1f5a0accaf7f9c7c418fa67aff70170c"
server_key_version: "encryptionkey@0"
I20260927 00:52:57.154361 5828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:52:57.157627 5842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:57.159479 5828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:52:57.159637 5828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "05d07d63053d41cb935b944f3dddc68e"
format_stamp: "Formatted at 2026-09-27 00:52:57 on dist-test-slave-9fh2"
server_key: "a7c2a36162063ebf2de8c7c399771979"
server_key_iv: "1f5a0accaf7f9c7c418fa67aff70170c"
server_key_version: "encryptionkey@0"
I20260927 00:52:57.159776 5828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:57.171444 5828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:57.172358 5828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:57.172591 5828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:57.182117 5828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41557
I20260927 00:52:57.182199 5894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41557 every 8 connection(s)
I20260927 00:52:57.183404 5828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:57.184645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5828
I20260927 00:52:57.184796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:57.185120 392 external_mini_cluster.cc:1428] Setting key 8de8894b482c149507c2ede9b35d3353
I20260927 00:52:57.187479 5895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:57.194523 5895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e: Bootstrap starting.
I20260927 00:52:57.197012 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:57.186913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.98.62:41557)
0927 00:52:57.187814 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:57.187837 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:57.187889 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:52:57.188787 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:57.188810 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:57.189325 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:57.189726 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:57.190220 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:57.191733 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:57.192768 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:57.193473 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:57.193650 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:57.194705 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:57.194711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:57.194726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:52:57.195267 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:57.195342 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:52:57.195369 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:57.195476 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:57.195674 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:52:57.195985 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":205,"threads_started":1}
I20260927 00:52:57.197415 5895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:57.198205 5895 log.cc:824] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:57.200755 5895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e: No bootstrap required, opened a new log
I20260927 00:52:57.201119 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:52:57.204829 5895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d07d63053d41cb935b944f3dddc68e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41557 } }
I20260927 00:52:57.205092 5895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:57.205127 5895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d07d63053d41cb935b944f3dddc68e, State: Initialized, Role: FOLLOWER
I20260927 00:52:57.205525 5895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d07d63053d41cb935b944f3dddc68e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41557 } }
I20260927 00:52:57.205639 5895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:57.205698 5895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:57.205775 5895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:52:57.206705 5895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d07d63053d41cb935b944f3dddc68e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41557 } }
I20260927 00:52:57.207026 5895 leader_election.cc:302] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d07d63053d41cb935b944f3dddc68e; no voters:
I20260927 00:52:57.207347 5895 leader_election.cc:288] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:57.207580 5900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:57.207864 5900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [term 1 LEADER]: Becoming Leader. State: Replica: 05d07d63053d41cb935b944f3dddc68e, State: Running, Role: LEADER
I20260927 00:52:57.208282 5900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d07d63053d41cb935b944f3dddc68e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41557 } }
I20260927 00:52:57.208669 5895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:57.210443 5902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d07d63053d41cb935b944f3dddc68e. Latest consensus state: current_term: 1 leader_uuid: "05d07d63053d41cb935b944f3dddc68e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d07d63053d41cb935b944f3dddc68e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41557 } } }
I20260927 00:52:57.210631 5902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:57.210457 5901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d07d63053d41cb935b944f3dddc68e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d07d63053d41cb935b944f3dddc68e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41557 } } }
I20260927 00:52:57.210844 5901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e [sys.catalog]: This master's current role is: LEADER
W20260927 00:52:57.214936 5915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:52:57.215031 5915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:52:57.215106 5913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:57.215905 5913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:57.223027 5913 catalog_manager.cc:1347] Generated new cluster ID: 1d1802ae312348fe93ca6df64e89b360
I20260927 00:52:57.223129 5913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:57.242264 5913 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:57.244014 5913 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:57.254325 5913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d07d63053d41cb935b944f3dddc68e: Generated new TSK 0
I20260927 00:52:57.255738 5913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:52:57.319713 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:57.266967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.0.98.62:41557)
0927 00:52:57.267138 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:57.267168 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:57.267330 (+ 162us) server_negotiation.cc:407] Connection header received
0927 00:52:57.267536 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:57.267540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:57.267618 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:57.267803 (+ 185us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:52:57.268086 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:57.269056 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:57.270033 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:57.270398 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:57.270529 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:57.271013 (+ 484us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:52:57.319027 (+ 48014us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:52:57.319509 (+ 482us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55424: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:52:57.319757 5924 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:57.266841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41557 (local address 127.0.0.1:55424)
0927 00:52:57.267232 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:57.267245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:57.267357 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:57.267837 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:57.267839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:57.267846 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:52:57.267963 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:57.267969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:57.269205 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:57.269208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:57.269911 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:57.269919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:57.270561 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:57.270588 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:57.270695 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:57.270739 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:52:57.319366 (+ 48627us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:52:57.319571 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
W20260927 00:52:57.319834 5898 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55424: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 00:52:57.320711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5828
2026-09-27T00:52:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (363 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (143 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:57 dist-test-slave-9fh2 krb5kdc[5936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:57 dist-test-slave-9fh2 krb5kdc[5936](info): set up 2 sockets
Sep 27 00:52:57 dist-test-slave-9fh2 krb5kdc[5936](info): commencing operation
krb5kdc: starting...
W20260927 00:52:58.555709 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:52:58 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:52:58.690560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39251
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:58.800460 5949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:58.800724 5949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:58.800763 5949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:58.804823 5949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:52:58.804899 5949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:58.804919 5949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:58.804934 5949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:52:58.804949 5949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:52:58.810195 5949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:58.811241 5949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:58.812247 5949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:58.821660 5954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:58.821673 5957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:58.821713 5955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:58.822240 5949 server_base.cc:1034] running on GCE node
I20260927 00:52:58.822839 5949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:58.824045 5949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:58.825220 5949 hybrid_clock.cc:648] HybridClock initialized: now 1790470378825197 us; error 48 us; skew 500 ppm
Sep 27 00:52:58 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:58.828351 5949 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:52:58.829499 5949 webserver.cc:460] Webserver started at http://127.0.98.62:46029/ using document root <none> and password file <none>
I20260927 00:52:58.830139 5949 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:58.830200 5949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:58.830407 5949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:58.833057 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c299aff4d5e04e3aa5d14ee979d972a9"
format_stamp: "Formatted at 2026-09-27 00:52:58 on dist-test-slave-9fh2"
server_key: "76cdfb35b73e468c8c652629448f4e0a"
server_key_iv: "e15c105c1b183997ae8923f1931b5454"
server_key_version: "encryptionkey@0"
I20260927 00:52:58.833776 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c299aff4d5e04e3aa5d14ee979d972a9"
format_stamp: "Formatted at 2026-09-27 00:52:58 on dist-test-slave-9fh2"
server_key: "76cdfb35b73e468c8c652629448f4e0a"
server_key_iv: "e15c105c1b183997ae8923f1931b5454"
server_key_version: "encryptionkey@0"
I20260927 00:52:58.838414 5949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:52:58.841482 5964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:58.843070 5949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:52:58.843259 5949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c299aff4d5e04e3aa5d14ee979d972a9"
format_stamp: "Formatted at 2026-09-27 00:52:58 on dist-test-slave-9fh2"
server_key: "76cdfb35b73e468c8c652629448f4e0a"
server_key_iv: "e15c105c1b183997ae8923f1931b5454"
server_key_version: "encryptionkey@0"
I20260927 00:52:58.843422 5949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:58.865145 5949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:58.868988 5949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:58.869176 5949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:58.877424 5949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39251
I20260927 00:52:58.877715 6016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39251 every 8 connection(s)
I20260927 00:52:58.878644 5949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:52:58.881747 6017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:52:58.887409 6017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9: Bootstrap starting.
I20260927 00:52:58.888514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 5949
I20260927 00:52:58.888641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:52:58.888979 392 external_mini_cluster.cc:1428] Setting key 5ce7d11f9d146ca6a64f0c036ea56420
I20260927 00:52:58.890033 6017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9: Neither blocks nor log segments found. Creating new log.
I20260927 00:52:58.890959 6017 log.cc:824] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:52:58.893374 6017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9: No bootstrap required, opened a new log
I20260927 00:52:58.897476 6017 raft_consensus.cc:348] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39251 } }
I20260927 00:52:58.897703 6017 raft_consensus.cc:374] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:52:58.897737 6017 raft_consensus.cc:729] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c299aff4d5e04e3aa5d14ee979d972a9, State: Initialized, Role: FOLLOWER
I20260927 00:52:58.898168 6017 consensus_queue.cc:260] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39251 } }
I20260927 00:52:58.898309 6017 raft_consensus.cc:388] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:52:58.898375 6017 raft_consensus.cc:482] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:52:58.898469 6017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:52:58 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:58.899435 6017 raft_consensus.cc:504] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39251 } }
I20260927 00:52:58.899765 6017 leader_election.cc:302] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c299aff4d5e04e3aa5d14ee979d972a9; no voters:
I20260927 00:52:58.900074 6017 leader_election.cc:288] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:52:58.900189 6022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:52:58.900390 6022 raft_consensus.cc:686] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [term 1 LEADER]: Becoming Leader. State: Replica: c299aff4d5e04e3aa5d14ee979d972a9, State: Running, Role: LEADER
I20260927 00:52:58.900689 6022 consensus_queue.cc:237] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39251 } }
I20260927 00:52:58.901386 6017 sys_catalog.cc:564] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:52:58.902422 6023 sys_catalog.cc:455] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39251 } } }
I20260927 00:52:58.902611 6023 sys_catalog.cc:458] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:58.902967 6024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c299aff4d5e04e3aa5d14ee979d972a9. Latest consensus state: current_term: 1 leader_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c299aff4d5e04e3aa5d14ee979d972a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39251 } } }
I20260927 00:52:58.903082 6024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:52:58.903560 6031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:52:58.906322 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:58.890415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.98.62:39251)
0927 00:52:58.890958 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:58.890979 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:58.891025 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:52:58.891814 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:58.891834 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:58.892225 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:58.892595 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:58.893481 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:58.894980 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:58.896040 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:58.896787 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:58.896959 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:58.900037 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:58.900044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:58.900067 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:58.900109 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:58.902320 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:58.903015 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:58.903022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:58.903031 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:58.903303 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:58.903643 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:58.903647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:58.903649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:58.904495 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:58.904745 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:58.905111 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 00:52:58.905516 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":368,"thread_start_us":148,"threads_started":1}
I20260927 00:52:58.909165 6031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:52:58.915558 6031 catalog_manager.cc:1347] Generated new cluster ID: dea70d81a59a434fbe8ed92305ba4158
I20260927 00:52:58.915638 6031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:52:58.925800 6031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:52:58.926774 6031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:52:58.937309 6031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c299aff4d5e04e3aa5d14ee979d972a9: Generated new TSK 0
I20260927 00:52:58.938011 6031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:52:58.974638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39251
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:59.086632 6043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:59.086917 6043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:59.086961 6043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:59.091035 6043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:59.091106 6043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:59.091188 6043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:52:59.096622 6043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:59.097729 6043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:59.098750 6043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:59.109079 6049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:59.109079 6048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:59.109095 6051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:59.109469 6043 server_base.cc:1034] running on GCE node
I20260927 00:52:59.110033 6043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:59.110915 6043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:59.112124 6043 hybrid_clock.cc:648] HybridClock initialized: now 1790470379112116 us; error 56 us; skew 500 ppm
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.115376 6043 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:52:59.116475 6043 webserver.cc:460] Webserver started at http://127.0.98.1:38345/ using document root <none> and password file <none>
I20260927 00:52:59.117139 6043 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:59.117202 6043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:59.117401 6043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:59.119895 6043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "c58ea028f71e478ea462aee2ce666162"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "80c515ca5f4158bd3943f276cfac223a"
server_key_iv: "0ab2f376582ef9c0c5ee13d46c381e7a"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.120515 6043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c58ea028f71e478ea462aee2ce666162"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "80c515ca5f4158bd3943f276cfac223a"
server_key_iv: "0ab2f376582ef9c0c5ee13d46c381e7a"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.124979 6043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:52:59.128005 6058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.129650 6043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:52:59.129815 6043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "c58ea028f71e478ea462aee2ce666162"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "80c515ca5f4158bd3943f276cfac223a"
server_key_iv: "0ab2f376582ef9c0c5ee13d46c381e7a"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.129947 6043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:59.162930 6043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:59.167392 6043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:59.167630 6043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:59.168443 6043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:59.169541 6043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:59.169596 6043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.169655 6043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:59.169677 6043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.180665 6043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35451
I20260927 00:52:59.180776 6171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35451 every 8 connection(s)
I20260927 00:52:59.181849 6043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:52:59.191586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6043
I20260927 00:52:59.191710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:52:59.192065 392 external_mini_cluster.cc:1428] Setting key aaef3fe0756b72971369d85ce5860810
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:59.197918 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.184204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55875 (local address 127.0.98.62:39251)
0927 00:52:59.184390 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.184395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.185545 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 00:52:59.187117 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.187121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.187212 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.187300 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.188655 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.189193 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.190365 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.190593 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.190677 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.193730 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:59.193734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:59.193736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:59.193761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:59.195190 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.195769 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.195773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.195775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.195851 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.196372 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.196377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.196379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.196672 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:59.196798 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:59.197497 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 00:52:59.197695 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:52:59.198663 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.184705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39251 (local address 127.0.98.1:55875)
0927 00:52:59.185353 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:59.185410 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:59.186872 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:59.187514 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:59.187522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:59.187952 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:59.188497 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:59.188512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.189327 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.189330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:59.190246 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:59.190256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.190752 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.190756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:59.190910 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.191809 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:59.191831 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:59.193557 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:59.195300 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:59.195306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:59.195323 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:59.195673 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:59.195953 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:59.195956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:59.195958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:59.196146 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:59.196814 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:59.196821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:59.197201 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 00:52:59.197401 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 00:52:59.197683 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":192,"threads_started":1}
I20260927 00:52:59.200130 6172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39251
I20260927 00:52:59.200385 6172 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:59.200934 6172 heartbeater.cc:511] Master 127.0.98.62:39251 requested a full tablet report, sending...
I20260927 00:52:59.202618 5977 ts_manager.cc:194] Registered new tserver with Master: c58ea028f71e478ea462aee2ce666162 (127.0.98.1:35451)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:52:59.203965 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:52:59.224285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39251
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:59.336673 6177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:59.336966 6177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:59.337034 6177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:59.341158 6177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:59.341235 6177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:59.341296 6177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:52:59.346858 6177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:59.348038 6177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:59.349256 6177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:59.359736 6185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:59.359684 6183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:59.359709 6182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:59.360172 6177 server_base.cc:1034] running on GCE node
I20260927 00:52:59.360754 6177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:59.361466 6177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:59.362646 6177 hybrid_clock.cc:648] HybridClock initialized: now 1790470379362616 us; error 58 us; skew 500 ppm
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.365535 6177 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:52:59.366531 6177 webserver.cc:460] Webserver started at http://127.0.98.2:39353/ using document root <none> and password file <none>
I20260927 00:52:59.367102 6177 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:59.367218 6177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:59.367416 6177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:59.369484 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4e4189965ed498bbceff24671c8e5e7"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "30d7378e74d2862be9335d22ebdabaa3"
server_key_iv: "f803353b7cdaac1e194afc6d28802ff0"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.369998 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4e4189965ed498bbceff24671c8e5e7"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "30d7378e74d2862be9335d22ebdabaa3"
server_key_iv: "f803353b7cdaac1e194afc6d28802ff0"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.374401 6177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:52:59.377552 6192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.379329 6177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:59.379474 6177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "e4e4189965ed498bbceff24671c8e5e7"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "30d7378e74d2862be9335d22ebdabaa3"
server_key_iv: "f803353b7cdaac1e194afc6d28802ff0"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.379601 6177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:59.407111 6177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:59.410282 6177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:59.410446 6177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:59.411006 6177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:59.411939 6177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:59.411993 6177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.412034 6177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:59.412053 6177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.422371 6177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33009
I20260927 00:52:59.422510 6305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33009 every 8 connection(s)
I20260927 00:52:59.423458 6177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:52:59.432747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6177
I20260927 00:52:59.432883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:52:59.433216 392 external_mini_cluster.cc:1428] Setting key 1afd1da45ef8ac01c3197708c1f09089
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:59.440529 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.425896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47919 (local address 127.0.98.62:39251)
0927 00:52:59.426070 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.426074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.427031 (+ 957us) server_negotiation.cc:407] Connection header received
0927 00:52:59.428795 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.428799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.428882 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.428970 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.430543 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.431121 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.432374 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.432629 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.432720 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.435982 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:59.435986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:59.435989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:59.436019 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:59.437729 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.438461 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.438464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.438465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.438565 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.438953 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.438955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.438956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.439267 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:59.439355 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:59.440142 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 00:52:59.440312 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:52:59.442631 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.426258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39251 (local address 127.0.98.2:47919)
0927 00:52:59.426820 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:59.426891 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:59.428524 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:59.429210 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:59.429219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:59.429703 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:59.430379 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:59.430398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.431264 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.431267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:59.432256 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:59.432266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.432796 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.432800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:59.433030 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.433794 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:59.433816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:59.435776 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:59.437881 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:59.437889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:59.437913 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:59.438320 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:59.438654 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:59.438656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:59.438658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:59.438848 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:59.439367 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:59.439375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:59.439788 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 00:52:59.440773 (+ 985us) client_negotiation.cc:240] Negotiation successful
0927 00:52:59.441239 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260927 00:52:59.444089 6306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39251
I20260927 00:52:59.444424 6306 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:52:59.445111 6306 heartbeater.cc:511] Master 127.0.98.62:39251 requested a full tablet report, sending...
I20260927 00:52:59.446341 5977 ts_manager.cc:194] Registered new tserver with Master: e4e4189965ed498bbceff24671c8e5e7 (127.0.98.2:33009)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:52:59.447191 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:52:59.467619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39251
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:52:59.601202 6311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:52:59.601564 6311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:52:59.601632 6311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:52:59.606105 6311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:52:59.606197 6311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:52:59.606297 6311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:52:59.612257 6311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:52:59.613739 6311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:52:59.615005 6311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:52:59.626814 6319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:52:59.626861 6317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:59.627207 6311 server_base.cc:1034] running on GCE node
W20260927 00:52:59.626878 6316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:52:59.627772 6311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:52:59.628692 6311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:52:59.629887 6311 hybrid_clock.cc:648] HybridClock initialized: now 1790470379629847 us; error 58 us; skew 500 ppm
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.633617 6311 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:52:59.634858 6311 webserver.cc:460] Webserver started at http://127.0.98.3:36249/ using document root <none> and password file <none>
I20260927 00:52:59.635634 6311 fs_manager.cc:359] Metadata directory not provided
I20260927 00:52:59.635710 6311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:52:59.635949 6311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:52:59.638556 6311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "1984807a5a6749d98cfb240b03ca2f43"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "18d0d0771a4ef432d76b3f707be3465e"
server_key_iv: "77cbb88cfd708c2f7cc95f5124826746"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.639263 6311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1984807a5a6749d98cfb240b03ca2f43"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "18d0d0771a4ef432d76b3f707be3465e"
server_key_iv: "77cbb88cfd708c2f7cc95f5124826746"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.644033 6311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:52:59.647286 6326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.649011 6311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:52:59.649183 6311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "1984807a5a6749d98cfb240b03ca2f43"
format_stamp: "Formatted at 2026-09-27 00:52:59 on dist-test-slave-9fh2"
server_key: "18d0d0771a4ef432d76b3f707be3465e"
server_key_iv: "77cbb88cfd708c2f7cc95f5124826746"
server_key_version: "encryptionkey@0"
I20260927 00:52:59.649323 6311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:52:59.671469 6311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:52:59.674652 6311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:52:59.674814 6311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:52:59.675462 6311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:52:59.676348 6311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:52:59.676399 6311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.676451 6311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:52:59.676481 6311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:52:59.685876 6311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46833
I20260927 00:52:59.686038 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46833 every 8 connection(s)
I20260927 00:52:59.686860 6311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:52:59.696086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6311
I20260927 00:52:59.696228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:52:59.696589 392 external_mini_cluster.cc:1428] Setting key 32fafa5d3064de18fd41155a51c96c74
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:52:59.702688 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.689273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46629 (local address 127.0.98.62:39251)
0927 00:52:59.689489 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.689495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.690636 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 00:52:59.692184 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.692188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.692251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.692356 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.693631 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.694248 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.695482 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.695727 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.695820 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.698781 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:59.698785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:59.698788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:59.698821 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:59.700443 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.700984 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.700987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.700988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.701073 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.701457 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.701460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.701461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.701690 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:59.701796 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:59.702393 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 00:52:59.702528 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 00:52:59.703673 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.689759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39251 (local address 127.0.98.3:46629)
0927 00:52:59.690439 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 00:52:59.690496 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:52:59.691913 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:52:59.692547 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:52:59.692554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:52:59.692928 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:52:59.693468 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:59.693481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.694364 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.694366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:59.695342 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:52:59.695353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.695894 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.695899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:52:59.696106 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.696908 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:52:59.696928 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:52:59.698585 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:52:59.700560 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:59.700565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:59.700579 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:59.700885 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:59.701149 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:52:59.701151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:52:59.701152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:52:59.701342 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:52:59.701808 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:52:59.701814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:52:59.702092 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 00:52:59.702309 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 00:52:59.702625 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":159,"threads_started":1}
I20260927 00:52:59.704949 6440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39251
I20260927 00:52:59.705325 6440 heartbeater.cc:464] Registering TS with master...
I20260927 00:52:59.706008 6440 heartbeater.cc:511] Master 127.0.98.62:39251 requested a full tablet report, sending...
I20260927 00:52:59.707331 5977 ts_manager.cc:194] Registered new tserver with Master: 1984807a5a6749d98cfb240b03ca2f43 (127.0.98.3:46833)
I20260927 00:52:59.707916 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46629
I20260927 00:52:59.711783 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/test-admin.keytab.
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.728758 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 00:52:59.731851 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:59.730134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.98.1:35451)
0927 00:52:59.730548 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.730555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.730587 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:52:59.730655 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.730662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.730919 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.731089 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.731662 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49826: BlockingRecv error: recv got EOF from 127.0.0.1:49826 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
W20260927 00:52:59.732311 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc)
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.733325 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.734218 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:59.744616 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.735194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.0.98.1:35451)
0927 00:52:59.735370 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.735373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.735603 (+ 230us) server_negotiation.cc:407] Connection header received
0927 00:52:59.735754 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.735757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.735822 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.735908 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.736465 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.737317 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.738204 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.738709 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.738799 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.741013 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:59.741023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:59.741042 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:59.741071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:59.742802 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.743326 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.743333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.743334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.743424 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.743657 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.743659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.743661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.743927 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:59.744109 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:59.744254 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 00:52:59.744463 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 00:52:59.749336 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:59.748261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.0.98.1:35451)
0927 00:52:59.748422 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.748427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.748501 (+ 74us) server_negotiation.cc:407] Connection header received
0927 00:52:59.748643 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.748647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.748699 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.748787 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.749202 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49844: BlockingRecv error: recv got EOF from 127.0.0.1:49844 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 00:52:59.749464 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.750434 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.751310 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:59.761040 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.752293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.0.98.1:35451)
0927 00:52:59.752423 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.752427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.752495 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:52:59.752661 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.752664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.752702 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.752774 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.753414 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.754029 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.754942 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.755259 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.755434 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.757617 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:59.757621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:59.757623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:59.757654 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:59.759100 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.759600 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.759603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.759605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.759700 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.760032 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.760036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.760037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.760260 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:59.760474 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:59.760732 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:52:59.760883 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:52:59.764514 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:52:59.763328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.0.98.1:35451)
0927 00:52:59.763501 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.763506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.763653 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:52:59.763800 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.763804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.763873 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.763940 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.764392 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49862: BlockingRecv error: recv got EOF from 127.0.0.1:49862 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 00:52:59.764871 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470348013064-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.765887 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:52:59.766744 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:52:59.775779 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:52:59.767757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.0.98.1:35451)
0927 00:52:59.767945 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:52:59.767948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:52:59.767968 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:52:59.768104 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:52:59.768106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:52:59.768146 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:52:59.768219 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:52:59.768748 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.769365 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.770202 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:52:59.770457 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:52:59.770554 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:52:59.772688 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:52:59.772691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:52:59.772694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:52:59.772717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:52:59.774118 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.774515 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.774518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.774519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.774608 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:52:59.774896 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:52:59.774899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:52:59.774900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:52:59.775085 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:52:59.775264 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 00:52:59.775470 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:52:59.775592 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:52:59.776971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6043
I20260927 00:52:59.782718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6177
I20260927 00:52:59.789338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6311
I20260927 00:52:59.794932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 5949
2026-09-27T00:52:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2200 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[6460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[6460](info): set up 2 sockets
Sep 27 00:52:59 dist-test-slave-9fh2 krb5kdc[6460](info): commencing operation
krb5kdc: starting...
W20260927 00:53:00.775686 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:53:00 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:53:00.906675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42173
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:01.035334 6473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:01.035677 6473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:01.035737 6473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:01.040050 6473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:01.040139 6473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:01.040169 6473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:01.040189 6473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:01.040208 6473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:01.046245 6473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:01.047639 6473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:01.048900 6473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:01.058723 6478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:01.058770 6479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.059057 6473 server_base.cc:1034] running on GCE node
W20260927 00:53:01.058923 6481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.059968 6473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:01.061357 6473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:01.062592 6473 hybrid_clock.cc:648] HybridClock initialized: now 1790470381062560 us; error 53 us; skew 500 ppm
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:01.065886 6473 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:53:01.066923 6473 webserver.cc:460] Webserver started at http://127.0.98.62:37979/ using document root <none> and password file <none>
I20260927 00:53:01.067610 6473 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:01.067687 6473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:01.067906 6473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:01.070210 6473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c9aacfc0faad4d6b81ae8968aeaadc07"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "7ca5d94b30b940dd818f4a860293178b"
server_key_iv: "6086cbe12860df1f97a4c19c1446e49f"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.070889 6473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9aacfc0faad4d6b81ae8968aeaadc07"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "7ca5d94b30b940dd818f4a860293178b"
server_key_iv: "6086cbe12860df1f97a4c19c1446e49f"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.075421 6473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 00:53:01.078516 6488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.080255 6473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:01.080425 6473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c9aacfc0faad4d6b81ae8968aeaadc07"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "7ca5d94b30b940dd818f4a860293178b"
server_key_iv: "6086cbe12860df1f97a4c19c1446e49f"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.080546 6473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:01.093595 6473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:01.096807 6473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:01.097010 6473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:01.104498 6473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42173
I20260927 00:53:01.104938 6540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42173 every 8 connection(s)
I20260927 00:53:01.105557 6473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:01.109365 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:01.114516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6473
I20260927 00:53:01.114661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:53:01.114946 392 external_mini_cluster.cc:1428] Setting key 568ff3611a936af7aba560ac28b93da1
I20260927 00:53:01.117010 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Bootstrap starting.
I20260927 00:53:01.120146 6541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:01.121121 6541 log.cc:824] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:01.123553 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: No bootstrap required, opened a new log
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:01.128603 6541 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 00:53:01.128963 6541 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:01.129029 6541 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9aacfc0faad4d6b81ae8968aeaadc07, State: Initialized, Role: FOLLOWER
I20260927 00:53:01.129616 6541 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 00:53:01.129804 6541 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:01.129880 6541 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:01.129985 6541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:01.131137 6541 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 00:53:01.131656 6541 leader_election.cc:302] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9aacfc0faad4d6b81ae8968aeaadc07; no voters:
I20260927 00:53:01.132043 6541 leader_election.cc:288] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:53:01.132148 6546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:01.132332 6546 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 LEADER]: Becoming Leader. State: Replica: c9aacfc0faad4d6b81ae8968aeaadc07, State: Running, Role: LEADER
I20260927 00:53:01.132652 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.116905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.98.62:42173)
0927 00:53:01.117416 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:01.117433 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:01.117471 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:53:01.118308 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:01.118330 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:01.118720 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:01.119277 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:01.120479 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.121602 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.122555 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.123091 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.123307 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.126398 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:01.126406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:01.126432 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:01.126476 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:01.128726 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.129449 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.129456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.129463 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.129643 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.130005 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.130011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.130012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.130624 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:01.130893 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:01.131191 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:53:01.131551 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":166,"threads_started":1}
I20260927 00:53:01.132694 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 00:53:01.133641 6541 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:01.134692 6548 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9aacfc0faad4d6b81ae8968aeaadc07. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } }
I20260927 00:53:01.134877 6548 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:01.135288 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } }
I20260927 00:53:01.135385 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:01.136209 6555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:01.140036 6555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:01.147037 6555 catalog_manager.cc:1347] Generated new cluster ID: d0d5a3fa083c4f2a9530f06c9f10ba31
I20260927 00:53:01.147136 6555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:01.168313 6555 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:53:01.169754 6555 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:01.178334 6555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Generated new TSK 0
I20260927 00:53:01.179347 6555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:53:01.223196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:01.366205 6567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:01.366492 6567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:01.366533 6567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:01.370690 6567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:01.370774 6567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:01.370834 6567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:01.376284 6567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:01.377444 6567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:01.378497 6567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:01.389473 6573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:01.389514 6572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.389772 6567 server_base.cc:1034] running on GCE node
W20260927 00:53:01.390141 6575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.390707 6567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:01.391472 6567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:01.392668 6567 hybrid_clock.cc:648] HybridClock initialized: now 1790470381392639 us; error 46 us; skew 500 ppm
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:01.395853 6567 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:53:01.396854 6567 webserver.cc:460] Webserver started at http://127.0.98.1:38639/ using document root <none> and password file <none>
I20260927 00:53:01.397436 6567 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:01.397508 6567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:01.397697 6567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:01.399838 6567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "d54f459a136143f59664ed20a1f2ea82"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "d757fb65f5d7db251f372603055be9b5"
server_key_iv: "d72a684e73ec1a087533094d602d92ce"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.400425 6567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d54f459a136143f59664ed20a1f2ea82"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "d757fb65f5d7db251f372603055be9b5"
server_key_iv: "d72a684e73ec1a087533094d602d92ce"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.404608 6567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 00:53:01.407473 6582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.408973 6567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:53:01.409107 6567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "d54f459a136143f59664ed20a1f2ea82"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "d757fb65f5d7db251f372603055be9b5"
server_key_iv: "d72a684e73ec1a087533094d602d92ce"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.409217 6567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:01.422191 6567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:01.425649 6567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:01.425854 6567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:01.426465 6567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:01.427362 6567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:01.427412 6567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.427469 6567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:01.427496 6567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.437182 6567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36749
I20260927 00:53:01.437242 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36749 every 8 connection(s)
I20260927 00:53:01.438143 6567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:01.442504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6567
I20260927 00:53:01.442618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:53:01.442896 392 external_mini_cluster.cc:1428] Setting key fd7dd14fdffdf10f351d0c292f71c39f
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:53:01.455401 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.440554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51325 (local address 127.0.98.62:42173)
0927 00:53:01.440739 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:01.440744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:01.441808 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 00:53:01.443414 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:01.443417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:01.443472 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:01.443582 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:01.445344 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.446123 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.447449 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.447728 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.447817 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.450982 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:01.450986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:01.450989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:01.451032 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:01.452874 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.453596 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.453600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.453602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.453689 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.454127 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.454130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.454132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.454356 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:01.454441 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:01.455072 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 00:53:01.455258 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 00:53:01.456781 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.440930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:51325)
0927 00:53:01.441594 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:01.441654 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:01.443103 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:01.443834 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:01.443844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:01.444451 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:01.445161 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:01.445177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.446268 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.446271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:01.447304 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:01.447314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.447958 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.447962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:01.448171 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.448874 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:01.448903 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:01.450773 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:01.453027 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:01.453034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:01.453050 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:01.453466 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:01.453826 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:01.453829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:01.453830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:01.453996 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:01.454467 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:01.454474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:01.454797 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 00:53:01.455016 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:53:01.455486 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":183,"threads_started":1}
I20260927 00:53:01.458267 6696 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 00:53:01.458546 6696 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:01.459185 6696 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 00:53:01.461177 6503 ts_manager.cc:194] Registered new tserver with Master: d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749)
I20260927 00:53:01.462335 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:53:01.475260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:01.610127 6701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:01.610622 6701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:01.610694 6701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:01.615288 6701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:01.615381 6701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:01.615473 6701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:01.621655 6701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:01.623135 6701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:01.624449 6701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:01.636282 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:01.636298 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:01.636298 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.636814 6701 server_base.cc:1034] running on GCE node
I20260927 00:53:01.637322 6701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:01.638136 6701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:01.639405 6701 hybrid_clock.cc:648] HybridClock initialized: now 1790470381639363 us; error 52 us; skew 500 ppm
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:01.646759 6701 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:53:01.648038 6701 webserver.cc:460] Webserver started at http://127.0.98.2:39637/ using document root <none> and password file <none>
I20260927 00:53:01.648733 6701 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:01.648803 6701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:01.649030 6701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:01.651575 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d021429bbf54df0bf1788a6017f91fd"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "46410834e548c160b6f4db01c0062f10"
server_key_iv: "59245b511d612003245f949e4f5b2b4a"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.652218 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d021429bbf54df0bf1788a6017f91fd"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "46410834e548c160b6f4db01c0062f10"
server_key_iv: "59245b511d612003245f949e4f5b2b4a"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.656917 6701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:53:01.659909 6716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.661685 6701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:01.661854 6701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "8d021429bbf54df0bf1788a6017f91fd"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "46410834e548c160b6f4db01c0062f10"
server_key_iv: "59245b511d612003245f949e4f5b2b4a"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.662020 6701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:01.678444 6701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:01.682641 6701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:01.682838 6701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:01.683609 6701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:01.684669 6701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:01.684736 6701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.684796 6701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:01.684826 6701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.695948 6701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41683
I20260927 00:53:01.695989 6829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41683 every 8 connection(s)
I20260927 00:53:01.697121 6701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:01.702996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6701
I20260927 00:53:01.703133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:53:01.703418 392 external_mini_cluster.cc:1428] Setting key 6c6b221ecf62eb4a9cdef12bea2c053a
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:01.714105 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.699358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36803 (local address 127.0.98.62:42173)
0927 00:53:01.699524 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:01.699528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:01.700554 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 00:53:01.702204 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:01.702208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:01.702271 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:01.702375 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:01.704105 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.704798 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.706198 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.706522 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.706644 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.709811 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:01.709814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:01.709817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:01.709849 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:01.711283 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.712013 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.712017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.712018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.712104 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.712585 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.712588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.712589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.712857 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:01.712951 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:01.713834 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 00:53:01.713974 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:53:01.715441 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.699720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:36803)
0927 00:53:01.700303 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:01.700384 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:01.701889 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:01.702611 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:01.702619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:01.703098 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:01.703935 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:01.703958 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.704954 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.704958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:01.706051 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:01.706063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.706709 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.706713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:01.706987 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.707707 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:01.707731 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:01.709580 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:01.711410 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:01.711418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:01.711441 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:01.711852 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:01.712203 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:01.712206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:01.712208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:01.712417 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:01.712984 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:01.712992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:01.713465 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 00:53:01.713741 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 00:53:01.714229 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260927 00:53:01.716972 6830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 00:53:01.717358 6830 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:01.717983 6830 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 00:53:01.719451 6503 ts_manager.cc:194] Registered new tserver with Master: 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2:41683)
I20260927 00:53:01.720199 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:53:01.735574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:01.852425 6835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:01.852708 6835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:01.852768 6835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:01.856784 6835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:01.856850 6835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:01.856933 6835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:01.862198 6835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:01.863332 6835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:01.864351 6835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:01.874451 6840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:01.874552 6843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.874734 6835 server_base.cc:1034] running on GCE node
W20260927 00:53:01.874452 6841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:01.875335 6835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:01.875991 6835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:01.877170 6835 hybrid_clock.cc:648] HybridClock initialized: now 1790470381877137 us; error 54 us; skew 500 ppm
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:01.879997 6835 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:53:01.880914 6835 webserver.cc:460] Webserver started at http://127.0.98.3:46495/ using document root <none> and password file <none>
I20260927 00:53:01.881471 6835 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:01.881538 6835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:01.881742 6835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:01.883817 6835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "23d56d004b164a5c892209495071c41b"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "fba098c652c6d0187f4a24d5b8a22783"
server_key_iv: "d23b0529e197cb99d6c6b872e3f9ddd5"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.884359 6835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23d56d004b164a5c892209495071c41b"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "fba098c652c6d0187f4a24d5b8a22783"
server_key_iv: "d23b0529e197cb99d6c6b872e3f9ddd5"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.888418 6835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:53:01.891327 6850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.892824 6835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:01.892949 6835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "23d56d004b164a5c892209495071c41b"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "fba098c652c6d0187f4a24d5b8a22783"
server_key_iv: "d23b0529e197cb99d6c6b872e3f9ddd5"
server_key_version: "encryptionkey@0"
I20260927 00:53:01.893057 6835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:01.910130 6835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:01.913496 6835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:01.913687 6835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:01.914291 6835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:01.915205 6835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:01.915257 6835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.915311 6835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:01.915340 6835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:01.924777 6835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39593
I20260927 00:53:01.924940 6963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39593 every 8 connection(s)
I20260927 00:53:01.925793 6835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:01.933455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6835
I20260927 00:53:01.933633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:53:01.933949 392 external_mini_cluster.cc:1428] Setting key d18ab2ec78ecfa3255600eff92880da9
Sep 27 00:53:01 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:01.942708 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.928211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47553 (local address 127.0.98.62:42173)
0927 00:53:01.928470 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:01.928475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:01.929458 (+ 983us) server_negotiation.cc:407] Connection header received
0927 00:53:01.931091 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:01.931095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:01.931213 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:01.931309 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:01.932868 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.933444 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.935002 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.935394 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.935526 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.938586 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:01.938589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:01.938592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:01.938622 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:01.940292 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.940856 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.940859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.940861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.940950 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:01.941427 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:01.941430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:01.941431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:01.941681 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:01.941797 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:01.942407 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 00:53:01.942541 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 00:53:01.943624 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:01.928625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:47553)
0927 00:53:01.929219 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:01.929293 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:01.930818 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:01.931596 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:01.931605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:01.932064 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:01.932699 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:01.932715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.933654 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.933659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:01.934859 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:01.934876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:01.935613 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:01.935618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:01.935878 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:01.936599 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:01.936616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:01.938387 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:01.940426 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:01.940431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:01.940445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:01.940740 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:01.941132 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:01.941135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:01.941136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:01.941292 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:01.941835 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:01.941841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:01.942144 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 00:53:01.942326 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:53:01.942621 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":134,"threads_started":1}
I20260927 00:53:01.945389 6964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 00:53:01.945686 6964 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:01.946224 6964 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 00:53:01.947613 6503 ts_manager.cc:194] Registered new tserver with Master: 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:01.948369 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47553
I20260927 00:53:01.948983 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 00:53:01.978566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33443
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42173,127.0.98.61:33443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:02.099500 6969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:02.099866 6969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:02.099917 6969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:02.104447 6969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:02.104549 6969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:02.104570 6969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:02.104588 6969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:02.104604 6969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:02.110750 6969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42173,127.0.98.61:33443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
W20260927 00:53:02.111054 6969 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42173,127.0.98.61:33443'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 00:53:02.112213 6969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:02.113479 6969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:02.123020 6975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.123003 6977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.123006 6974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.123481 6969 server_base.cc:1034] running on GCE node
I20260927 00:53:02.124171 6969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:02.125487 6969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:02.126663 6969 hybrid_clock.cc:648] HybridClock initialized: now 1790470382126633 us; error 54 us; skew 500 ppm
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:02.129827 6969 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:53:02.130951 6969 webserver.cc:460] Webserver started at http://127.0.98.61:40235/ using document root <none> and password file <none>
I20260927 00:53:02.131685 6969 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.131760 6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.131994 6969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:02.134482 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/instance:
uuid: "3c84ad0ddc974750ab34976f75f05296"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "7bfb8e2b88fcd5096896fd4d62eb72c5"
server_key_iv: "c91e62691ffd9f1a6f29bbd1d8dcbe37"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.135178 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal/instance:
uuid: "3c84ad0ddc974750ab34976f75f05296"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "7bfb8e2b88fcd5096896fd4d62eb72c5"
server_key_iv: "c91e62691ffd9f1a6f29bbd1d8dcbe37"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.139951 6969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:53:02.143082 6984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.144754 6969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:53:02.144927 6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
uuid: "3c84ad0ddc974750ab34976f75f05296"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "7bfb8e2b88fcd5096896fd4d62eb72c5"
server_key_iv: "c91e62691ffd9f1a6f29bbd1d8dcbe37"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.145061 6969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.162442 6969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:02.166262 6969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:02.166479 6969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:02.174849 6969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33443
I20260927 00:53:02.175191 7036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33443 every 8 connection(s)
I20260927 00:53:02.176191 6969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:53:02.179847 7037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:02.184612 7037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:53:02.186501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 6969
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:53:02.201793 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.186985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.0.98.62:42173)
0927 00:53:02.187180 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.187185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.188387 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 00:53:02.189607 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.189611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.189657 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.189773 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.190927 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.191763 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.193295 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.193658 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.193765 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.197177 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.197181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.197184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.197215 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.199180 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.199864 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.199867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.199868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.199958 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.200293 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.200296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.200297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.200681 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.200787 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.201281 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.201429 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 00:53:02.202745 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.187591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:36406)
0927 00:53:02.188174 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.188231 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.189339 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.189851 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.189856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.190369 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.190779 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.190796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.191930 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.191933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.193151 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.193163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.193820 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.193823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.194068 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.194910 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.194931 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.196977 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.199309 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.199320 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.199332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.199746 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.200054 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.200057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.200059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.200195 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.200807 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.200813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.201024 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.201225 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.201636 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":56,"spinlock_wait_cycles":4992,"thread_start_us":135,"threads_started":1}
I20260927 00:53:02.202755 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.188617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.98.61:33443)
0927 00:53:02.188979 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.188989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.189088 (+ 99us) server_negotiation.cc:407] Connection header received
0927 00:53:02.189338 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.189358 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.189627 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.189834 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.190786 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.192147 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.193298 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.193908 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.194141 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.197551 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.197559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.197573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.197595 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.199569 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.200067 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.200072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.200073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.200189 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.200467 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.200470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.200471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.201007 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.201169 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.201481 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.201700 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260927 00:53:02.204573 7037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:53:02.214177 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.205146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:41812)
0927 00:53:02.205416 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.205432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.205518 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.205802 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.205805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.206008 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.206126 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.206132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.207072 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.207076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.207894 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.207903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.208422 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.208426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.208522 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.209012 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.209024 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.210656 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.212490 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.212493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.212495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.212743 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.213045 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.213048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.213051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.213154 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.213568 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.213571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.213688 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.213790 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.213955 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 00:53:02.214314 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.205287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.0.98.61:33443)
0927 00:53:02.205420 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.205424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.205488 (+ 64us) server_negotiation.cc:407] Connection header received
0927 00:53:02.205601 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.205604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.205640 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.205719 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.206244 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.206927 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.208005 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.208289 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.208374 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.210832 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.210835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.210837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.210861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.212303 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.212851 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.212853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.212855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.212927 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.213251 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.213253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.213254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.213448 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.213538 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.213862 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.214024 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":46}
I20260927 00:53:02.218251 7037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Bootstrap starting.
I20260927 00:53:02.221256 7037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:02.222131 7037 log.cc:824] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:02.224772 7037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: No bootstrap required, opened a new log
I20260927 00:53:02.228405 7037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } }
I20260927 00:53:02.228668 7037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:02.228726 7037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c84ad0ddc974750ab34976f75f05296, State: Initialized, Role: FOLLOWER
I20260927 00:53:02.229207 7037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } }
I20260927 00:53:02.229835 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } } }
I20260927 00:53:02.230036 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:02.230690 7037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:53:02.236328 7054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:53:02.236438 7054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:53:02.239078 6969 master_runner.cc:428] Detected that this master 3c84ad0ddc974750ab34976f75f05296 is joining an existing cluster
I20260927 00:53:02.239163 6969 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33443
I20260927 00:53:02.250638 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.241436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.0.98.62:42173)
0927 00:53:02.241562 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.241567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.242013 (+ 446us) server_negotiation.cc:407] Connection header received
0927 00:53:02.242172 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.242175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.242225 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.242302 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.243134 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.243709 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.245067 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.245457 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.245590 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.246818 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.246822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.246826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.246854 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.248603 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.249261 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.249266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.249268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.249363 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.249701 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.249704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.249705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.249943 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.250051 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.250362 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.250502 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:53:02.251209 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.241365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:36418)
0927 00:53:02.241903 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.241914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.242040 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.242797 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.242799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.242927 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.243015 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.243030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.243817 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.243819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.244920 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.244926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.245629 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.245632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.245713 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.246279 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.246290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.246653 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.248728 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.248731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.248734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.249150 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.249509 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.249512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.249514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.249617 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.250090 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.250093 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.250184 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.250849 (+ 665us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.251005 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":113,"spinlock_wait_cycles":8320,"thread_start_us":91,"threads_started":1}
I20260927 00:53:02.251477 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.241788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.0.98.61:33443)
0927 00:53:02.241879 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.241883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.242064 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:53:02.242145 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.242148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.242208 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.242301 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.242792 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.243688 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.245741 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.246083 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.246198 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.247954 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.247957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.247959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.247978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.249643 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.250092 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.250096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.250098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.250170 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.250441 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.250445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.250447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.250712 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.250809 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.251112 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.251285 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":2304}
I20260927 00:53:02.252025 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.241521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:41822)
0927 00:53:02.241892 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.241908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.242040 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.242344 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.242347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.242552 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.242682 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.242688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.243815 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.243818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.244908 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.244925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.246316 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.246319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.246408 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.247291 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.247305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.247674 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.249738 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.249741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.249743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.249948 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.250268 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.250270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.250273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.250346 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.250848 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.250851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.250943 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.251691 (+ 748us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.251838 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":42,"spinlock_wait_cycles":29952,"thread_start_us":190,"threads_started":1}
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:53:02.265038 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.255710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.0.98.61:33443)
0927 00:53:02.256512 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.256516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.256531 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:02.256603 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.256607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.256660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.256745 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.257547 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.258374 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.259624 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.259851 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.259941 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.262098 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.262101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.262103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.262125 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.263443 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.263864 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.263867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.263868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.263933 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.264173 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.264176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.264178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.264349 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.264430 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.264748 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.264866 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722}
I20260927 00:53:02.265050 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.255676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:41834)
0927 00:53:02.256048 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.256095 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.256211 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.256785 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.256790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.257269 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.257407 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.257415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.258491 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.258494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.259494 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.259505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.259950 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.259953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.260054 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.260551 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.260563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.261948 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.263541 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.263550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.263557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.263776 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.264010 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.264012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.264014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.264099 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.264441 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.264449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.264564 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.264686 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.264871 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:53:02.266827 6505 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c84ad0ddc974750ab34976f75f05296"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33443
}
attrs {
promote: true
}
}
dest_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07"
cas_config_opid_index: -1
I20260927 00:53:02.267588 6505 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 00:53:02.269838 7011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:02.271088 7011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Refusing update from remote peer c9aacfc0faad4d6b81ae8968aeaadc07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 00:53:02.272007 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c84ad0ddc974750ab34976f75f05296 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.272822 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:02.274271 7066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33443
I20260927 00:53:02.274803 6969 master.cc:537] Master@127.0.98.61:33443 shutting down...
I20260927 00:53:02.278968 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.279093 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:02.278946 7011 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c84ad0ddc974750ab34976f75f05296 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.281121 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9aacfc0faad4d6b81ae8968aeaadc07. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.281219 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.281304 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.281383 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.281447 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.281507 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.283876 7069 mvcc.cc:205] Tried to move back new op lower bound from 7333766681301024768 to 7333766681120489472. Current Snapshot: MvccSnapshot[applied={T|T < 7333766681165889536}]
I20260927 00:53:02.284544 6969 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Raft consensus shutting down.
I20260927 00:53:02.284819 6969 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Raft consensus is shut down!
I20260927 00:53:02.284894 6969 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 00:53:02.291366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46263
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:53:02.301692 6969 master.cc:552] Master@127.0.98.61:33443 shutdown complete.
I20260927 00:53:02.301766 6969 master_runner.cc:305] Clearing existing system tablet
I20260927 00:53:02.302023 6969 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.302081 6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.303566 6969 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:53:02.304519 7079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.304858 6969 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.304991 6969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.305040 6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
uuid: "3c84ad0ddc974750ab34976f75f05296"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "7bfb8e2b88fcd5096896fd4d62eb72c5"
server_key_iv: "c91e62691ffd9f1a6f29bbd1d8dcbe37"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.306955 6969 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:02.309610 6969 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:53:02.309705 6969 log.cc:1196] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 00:53:02.309979 6969 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Deleting consensus metadata
I20260927 00:53:02.310149 6969 master_runner.cc:315] Copying system tablet from 127.0.98.62:42173
I20260927 00:53:02.311367 6969 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42173
I20260927 00:53:02.320686 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.312152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.0.98.62:42173)
0927 00:53:02.312287 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.312292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.312645 (+ 353us) server_negotiation.cc:407] Connection header received
0927 00:53:02.312840 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.312843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.312889 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.312939 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.313632 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.314395 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.315579 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.315881 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.315947 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.317369 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.317372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.317375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.317399 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.318929 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.319429 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.319432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.319434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.319514 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.319790 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.319793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.319795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.319998 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.320080 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.320404 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.320519 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:53:02.321280 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.312055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:36424)
0927 00:53:02.312529 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.312546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.312730 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.313089 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.313091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.313307 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.313507 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.313517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.314531 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.314534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.315449 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.315459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.316120 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.316124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.316213 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.316765 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.316779 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.317221 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.319065 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.319068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.319070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.319320 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.319613 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.319616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.319618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.319697 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.320144 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.320147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.320240 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.320914 (+ 674us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.321083 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":140,"threads_started":1}
I20260927 00:53:02.321928 6525 tablet_copy_service.cc:140] P c9aacfc0faad4d6b81ae8968aeaadc07: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c84ad0ddc974750ab34976f75f05296 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36424)
I20260927 00:53:02.322201 6525 tablet_copy_service.cc:161] P c9aacfc0faad4d6b81ae8968aeaadc07: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c84ad0ddc974750ab34976f75f05296 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36424: session id = 3c84ad0ddc974750ab34976f75f05296-00000000000000000000000000000000
I20260927 00:53:02.324492 6525 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:53:02.326529 6969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:53:02.328965 6969 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: tablet copy: Starting download of 0 data blocks...
I20260927 00:53:02.329165 6969 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: tablet copy: Starting download of 1 WAL segments...
I20260927 00:53:02.331336 6969 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:53:02.333369 6525 tablet_copy_service.cc:342] P c9aacfc0faad4d6b81ae8968aeaadc07: Request end of tablet copy session 3c84ad0ddc974750ab34976f75f05296-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36424
I20260927 00:53:02.333530 6525 tablet_copy_service.cc:433] P c9aacfc0faad4d6b81ae8968aeaadc07: ending tablet copy session 3c84ad0ddc974750ab34976f75f05296-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c84ad0ddc974750ab34976f75f05296
I20260927 00:53:02.342088 6969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:02.344244 7086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.344558 7087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.344663 6969 server_base.cc:1034] running on GCE node
W20260927 00:53:02.347321 7089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.347681 6969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:02.348064 6969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:02.349210 6969 hybrid_clock.cc:648] HybridClock initialized: now 1790470382349119 us; error 116 us; skew 500 ppm
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:02.350863 6969 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:53:02.351416 6969 webserver.cc:460] Webserver started at http://127.0.98.61:40597/ using document root <none> and password file <none>
I20260927 00:53:02.351601 6969 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.351645 6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.353072 6969 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.354261 7096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.354579 6969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.354656 6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
uuid: "3c84ad0ddc974750ab34976f75f05296"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "7bfb8e2b88fcd5096896fd4d62eb72c5"
server_key_iv: "c91e62691ffd9f1a6f29bbd1d8dcbe37"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.354727 6969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.364658 6969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:02.366850 6969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:02.371969 6969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33443
I20260927 00:53:02.372160 7148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33443 every 8 connection(s)
I20260927 00:53:02.372601 6969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:53:02.374861 7149 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:53:02.376183 7149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Bootstrap starting.
I20260927 00:53:02.385836 7149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:53:02.386228 7149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: Bootstrap complete.
I20260927 00:53:02.386938 7149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } }
I20260927 00:53:02.387086 7149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c84ad0ddc974750ab34976f75f05296, State: Initialized, Role: LEARNER
I20260927 00:53:02.387225 7149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } }
I20260927 00:53:02.387764 7149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:02.387691 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } }
I20260927 00:53:02.388015 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.393771 7163 catalog_manager.cc:1259] Loaded cluster ID: d0d5a3fa083c4f2a9530f06c9f10ba31
I20260927 00:53:02.393906 7163 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: loading cluster ID for follower catalog manager: success
I20260927 00:53:02.396376 7163 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: acquiring CA information for follower catalog manager: success
I20260927 00:53:02.398176 7163 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:53:02.430606 7077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:02.430896 7077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:02.430953 7077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:02.435319 7077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:02.435384 7077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:02.435408 7077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:02.435431 7077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:02.435461 7077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:02.442231 7077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:02.443332 7077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:02.444368 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:02.453446 7168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.453446 7171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.453469 7169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.453696 7077 server_base.cc:1034] running on GCE node
I20260927 00:53:02.454674 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:02.456233 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:02.457458 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790470382457405 us; error 50 us; skew 500 ppm
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:02.460960 7077 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:53:02.462165 7077 webserver.cc:460] Webserver started at http://127.0.98.60:41507/ using document root <none> and password file <none>
I20260927 00:53:02.462855 7077 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.462930 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.463217 7077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:02.466156 6696 heartbeater.cc:503] Master 127.0.98.62:42173 was elected leader, sending a full tablet report...
I20260927 00:53:02.465902 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/instance:
uuid: "aeaf33431e9d42a19945e33ba8c02055"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "b5e9ce0b0936dde78c69752a8dd78694"
server_key_iv: "b4afdba597699e516e05e37b248560c0"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.466606 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal/instance:
uuid: "aeaf33431e9d42a19945e33ba8c02055"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "b5e9ce0b0936dde78c69752a8dd78694"
server_key_iv: "b4afdba597699e516e05e37b248560c0"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.471479 7077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 00:53:02.474808 7178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.476573 7077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:02.476745 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
uuid: "aeaf33431e9d42a19945e33ba8c02055"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "b5e9ce0b0936dde78c69752a8dd78694"
server_key_iv: "b4afdba597699e516e05e37b248560c0"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.476891 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.503841 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:02.507838 7077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:02.508057 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:02.516640 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46263
I20260927 00:53:02.516934 7230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46263 every 8 connection(s)
I20260927 00:53:02.517899 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:53:02.518712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 7077
I20260927 00:53:02.521440 7231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:02.525811 7231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:02.535715 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.520384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.0.98.60:46263)
0927 00:53:02.520940 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.520959 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.521009 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:53:02.521910 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.521933 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.522385 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.522748 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.523784 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.524968 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.526185 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.526939 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.527110 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.530111 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.530118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.530129 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.530161 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.532041 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.532527 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.532532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.532538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.532663 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.532972 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.532977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.532978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.533440 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.533638 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.533970 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.534620 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":168,"threads_started":1}
I20260927 00:53:02.537762 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.528112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.98.62:42173)
0927 00:53:02.528274 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.528278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.528586 (+ 308us) server_negotiation.cc:407] Connection header received
0927 00:53:02.528763 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.528766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.528864 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.528959 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.529697 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.530227 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.531372 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.531594 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.531684 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.533968 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.533971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.533974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.533998 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.535611 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.536300 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.536304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.536305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.536405 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.536783 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.536786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.536787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.537018 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.537098 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.537514 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.537615 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:53:02.537958 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.527881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:36428)
0927 00:53:02.528418 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.528478 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.528636 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.529026 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.529031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.529389 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.529551 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.529559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.530357 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.530361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.531258 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.531269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.531709 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.531713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.531822 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.532277 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.532288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.533823 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.535839 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.535851 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.535858 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.536197 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.536585 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.536588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.536589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.536678 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.537123 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.537130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.537300 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.537471 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.537726 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":162,"threads_started":1}
I20260927 00:53:02.539239 7231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:53:02.548409 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.539833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:41844)
0927 00:53:02.539989 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.540005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.540091 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.540562 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.540565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.540717 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.540832 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.540837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.541833 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.541835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.542400 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.542406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.542856 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.542859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.542933 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.543491 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.543502 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.544961 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.546757 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.546761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.546763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.547008 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.547436 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.547438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.547440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.547520 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.547863 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.547865 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.548005 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.548094 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.548242 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 00:53:02.548498 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.539970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.0.98.61:33443)
0927 00:53:02.540289 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.540294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.540309 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:02.540357 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.540360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.540409 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.540525 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.540946 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.541729 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.542510 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.542751 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.542835 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.545191 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.545195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.545197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.545227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.546614 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.547248 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.547252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.547253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.547338 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.547595 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.547597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.547598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.547789 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.547868 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.548223 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.548347 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":104,"threads_started":1}
I20260927 00:53:02.549464 7231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:53:02.558135 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.550008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.0.1:58138)
0927 00:53:02.550143 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.550159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.550239 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.550537 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.550540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.550700 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.550799 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.550804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.551986 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.551989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.552566 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.552574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.553046 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.553048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.553110 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.553637 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.553648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.555098 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.556776 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.556780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.556782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.556952 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.557188 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.557190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.557192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.557263 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.557605 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.557608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.557704 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.557787 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.557941 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:53:02.558135 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.550180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.98.60:46263)
0927 00:53:02.550321 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.550325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.550338 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:53:02.550375 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.550378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.550417 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.550497 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.550909 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.551868 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.552676 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.552920 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.553004 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.555297 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.555300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.555302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.555321 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.556676 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.557045 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.557048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.557050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.557113 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.557334 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.557337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.557339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.557518 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.557604 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.557837 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.557995 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":55}
I20260927 00:53:02.561650 7231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Bootstrap starting.
I20260927 00:53:02.563994 7231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:02.564700 7231 log.cc:824] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:02.566910 7231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: No bootstrap required, opened a new log
I20260927 00:53:02.570360 7231 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } }
I20260927 00:53:02.570580 7231 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:02.570613 7231 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeaf33431e9d42a19945e33ba8c02055, State: Initialized, Role: FOLLOWER
I20260927 00:53:02.571036 7231 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } }
I20260927 00:53:02.571746 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } } }
I20260927 00:53:02.571978 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:02.572314 7231 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:53:02.578039 7248 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:53:02.578171 7248 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:53:02.580811 7077 master_runner.cc:428] Detected that this master aeaf33431e9d42a19945e33ba8c02055 is joining an existing cluster
I20260927 00:53:02.580883 7077 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46263
I20260927 00:53:02.589950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6567
I20260927 00:53:02.593986 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.583797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.0.1:58146)
0927 00:53:02.584186 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.584199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.584285 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.584595 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.584598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.585062 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.585372 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.585382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.586440 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.586443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.587542 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.587553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.588286 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.588451 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.589095 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.589111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.589613 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.592190 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.592194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.592196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.592482 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.592793 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.592796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.592798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.592889 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.593335 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.593338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.593452 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.593561 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.593717 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":241,"spinlock_wait_cycles":18816,"thread_start_us":123,"threads_started":1}
I20260927 00:53:02.594024 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.583851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.0.98.60:46263)
0927 00:53:02.583987 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.583990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.584287 (+ 297us) server_negotiation.cc:407] Connection header received
0927 00:53:02.584383 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.584386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.584421 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.584464 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.585495 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.586304 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.587661 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.588052 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.589754 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.589758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.589761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.589790 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.592058 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.592590 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.592594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.592596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.592696 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.592980 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.592983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.592985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.593240 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.593342 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.593639 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.593781 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":80}
I20260927 00:53:02.598448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36749
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42173
--webserver_port=38639
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:53:02.617363 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.583046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51320)
0927 00:53:02.583480 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.583496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.583663 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.584620 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.584622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.585086 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.585418 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.585425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.586437 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.586440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.587652 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.587660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.588286 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.588447 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.612500 (+ 24053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.612522 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.613236 (+ 714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.615448 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.615452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.615455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.615778 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.616077 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.616080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.616082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.616231 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.616707 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.616710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.616854 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.616951 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.617146 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":243,"spinlock_wait_cycles":22528,"thread_start_us":249,"threads_started":1}
I20260927 00:53:02.617960 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.583105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.98.62:42173)
0927 00:53:02.583240 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.583246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.583586 (+ 340us) server_negotiation.cc:407] Connection header received
0927 00:53:02.583893 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.583896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.583947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.583998 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.585543 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.586307 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.587755 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.588071 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588132 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.613495 (+ 25363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.613500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.613503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.613535 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.615295 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.615896 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.615900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.615902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.615987 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.616338 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.616341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.616343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.616577 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.616665 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.617662 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.617795 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:53:02.617410 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.584201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.0.98.61:33443)
0927 00:53:02.584340 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.584344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.584356 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:53:02.584389 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.584392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.584431 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.584514 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.585701 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.586627 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588070 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.588426 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588507 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.613369 (+ 24862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.613375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.613379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.613421 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.615203 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.615871 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.615875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.615877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.615967 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.616286 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.616290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.616292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.616558 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.616637 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.617033 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.617205 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:53:02.619903 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.583294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:34264)
0927 00:53:02.583665 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.583678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.583747 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.584564 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.584568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.585081 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.585518 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.585529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.586788 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.586792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.587935 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.587946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.588702 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.588705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.588804 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.612187 (+ 23383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.612215 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.613049 (+ 834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.615378 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.615382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.615385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.615752 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.616059 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.616072 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.616074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.616180 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.616692 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.616695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.616839 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.619462 (+ 2623us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.619674 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":41,"spinlock_wait_cycles":9984,"thread_start_us":155,"threads_started":1}
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470381, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:53:02.631484 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.621998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.0.1:58158)
0927 00:53:02.622245 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.622259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.622334 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.622708 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.622711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.622904 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.623023 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.623029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.624047 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.624050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.624935 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.624945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.625514 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.625518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.625627 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.626098 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.626122 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.627692 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.629730 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.629734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.629736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.629999 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.630324 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.630327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.630329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.630419 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.630864 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.630867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.630976 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.631089 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.631289 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 00:53:02.631907 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.622282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.0.98.60:46263)
0927 00:53:02.622414 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.622418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.622433 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:02.622517 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.622520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.622580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.622675 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.623141 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.623926 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.625061 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.625393 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.625502 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.627873 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.627877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.627880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.627912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.629602 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.630108 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.630112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.630114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.630219 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.630515 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.630518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.630520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.630769 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.630867 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.631188 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.631346 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:53:02.633177 6505 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46263: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46263
}
attrs {
promote: true
}
}
dest_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07"
cas_config_opid_index: 5
I20260927 00:53:02.633816 6505 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } }
I20260927 00:53:02.635599 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.637224 7066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46263
I20260927 00:53:02.637414 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.637503 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:02.638439 7205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:02.640022 7205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Refusing update from remote peer c9aacfc0faad4d6b81ae8968aeaadc07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:53:02.640522 7077 master.cc:537] Master@127.0.98.60:46263 shutting down...
I20260927 00:53:02.641415 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:53:02.642952 6490 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 00:53:02.644708 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.635274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:34266)
0927 00:53:02.639619 (+ 4345us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.639640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.639796 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.640341 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.640344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.640363 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:02.640529 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.640538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.641678 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.641682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.642958 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.642970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.644176 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.644180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.644293 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.644342 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.644417 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.644521 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4220}
I20260927 00:53:02.645354 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.635468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.98.61:33443)
0927 00:53:02.635647 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.635653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.639908 (+ 4255us) server_negotiation.cc:407] Connection header received
0927 00:53:02.639985 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.639989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.640073 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.640163 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:02.640685 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.641536 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.643106 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.643827 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.643939 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.644108 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.644914 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.645186 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:53:02.646353 7123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Refusing update from remote peer c9aacfc0faad4d6b81ae8968aeaadc07: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260927 00:53:02.647275 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 -> Peer aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60:46263): Couldn't send request to peer aeaf33431e9d42a19945e33ba8c02055. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:53:02.647645 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 00:53:02.649118 7123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.651203 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9aacfc0faad4d6b81ae8968aeaadc07. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.651327 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.651415 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: true } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.651475 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.651522 7065 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: attempting to promote NON_VOTER 3c84ad0ddc974750ab34976f75f05296 to VOTER
I20260927 00:53:02.652086 7077 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:02.652103 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } }
I20260927 00:53:02.652411 7077 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:02.652505 7077 tablet_replica.cc:330] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: stopping tablet replica
I20260927 00:53:02.654246 7123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 LEARNER]: Refusing update from remote peer c9aacfc0faad4d6b81ae8968aeaadc07: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 00:53:02.654801 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 00:53:02.654994 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 -> Peer aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60:46263): Couldn't send request to peer aeaf33431e9d42a19945e33ba8c02055. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:53:02.656821 7068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c84ad0ddc974750ab34976f75f05296 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.658658 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.658769 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:02.658891 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.658951 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:02.660050 7122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c84ad0ddc974750ab34976f75f05296 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.662226 7260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.662676 7260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:02.673273 7077 master.cc:552] Master@127.0.98.60:46263 shutdown complete.
I20260927 00:53:02.673359 7077 master_runner.cc:305] Clearing existing system tablet
I20260927 00:53:02.673641 7077 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.673704 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.675341 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.676425 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.676764 7077 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.676911 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.676959 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
uuid: "aeaf33431e9d42a19945e33ba8c02055"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "b5e9ce0b0936dde78c69752a8dd78694"
server_key_iv: "b4afdba597699e516e05e37b248560c0"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.679466 7077 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:02.681957 7077 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:53:02.682057 7077 log.cc:1196] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 00:53:02.682315 7077 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Deleting consensus metadata
I20260927 00:53:02.682478 7077 master_runner.cc:315] Copying system tablet from 127.0.98.62:42173
I20260927 00:53:02.683712 7077 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42173
I20260927 00:53:02.694120 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.684508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.0.98.62:42173)
0927 00:53:02.684661 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.684666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.685034 (+ 368us) server_negotiation.cc:407] Connection header received
0927 00:53:02.685199 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.685203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.685282 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.685403 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.686058 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.686845 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.688223 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.688622 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.688741 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.690352 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.690356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.690359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.690394 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.692082 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.692658 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.692661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.692663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.692776 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.693100 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.693103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.693105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.693353 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.693455 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.693785 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.693923 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:53:02.694155 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.684463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51322)
0927 00:53:02.684887 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.684905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.685085 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.685451 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.685454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.685736 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.685912 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.685919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.686993 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.686997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.687956 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.687967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.688761 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.688765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.688859 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.689529 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.689544 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.690162 (+ 618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.692231 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.692235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.692237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.692541 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.692881 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.692884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.692886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.692989 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.693458 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.693461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.693597 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.693710 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.693911 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":123,"threads_started":1}
I20260927 00:53:02.695186 6525 tablet_copy_service.cc:140] P c9aacfc0faad4d6b81ae8968aeaadc07: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aeaf33431e9d42a19945e33ba8c02055 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51322)
I20260927 00:53:02.695302 6525 tablet_copy_service.cc:161] P c9aacfc0faad4d6b81ae8968aeaadc07: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aeaf33431e9d42a19945e33ba8c02055 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51322: session id = aeaf33431e9d42a19945e33ba8c02055-00000000000000000000000000000000
I20260927 00:53:02.696524 6525 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:53:02.700565 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:53:02.703007 7077 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: tablet copy: Starting download of 0 data blocks...
I20260927 00:53:02.703274 7077 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: tablet copy: Starting download of 1 WAL segments...
I20260927 00:53:02.705677 7077 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:53:02.707872 6525 tablet_copy_service.cc:342] P c9aacfc0faad4d6b81ae8968aeaadc07: Request end of tablet copy session aeaf33431e9d42a19945e33ba8c02055-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51322
I20260927 00:53:02.707981 6525 tablet_copy_service.cc:433] P c9aacfc0faad4d6b81ae8968aeaadc07: ending tablet copy session aeaf33431e9d42a19945e33ba8c02055-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aeaf33431e9d42a19945e33ba8c02055
I20260927 00:53:02.718850 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:02.721036 7272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.721587 7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.721874 7077 server_base.cc:1034] running on GCE node
I20260927 00:53:02.723054 6830 heartbeater.cc:503] Master 127.0.98.62:42173 was elected leader, sending a full tablet report...
W20260927 00:53:02.723309 7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.723802 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:02.724174 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:02.725318 7077 hybrid_clock.cc:648] HybridClock initialized: now 1790470382725307 us; error 32 us; skew 500 ppm
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:02.727250 7077 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:53:02.727813 7077 webserver.cc:460] Webserver started at http://127.0.98.60:39415/ using document root <none> and password file <none>
I20260927 00:53:02.728034 7077 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.728080 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.729669 7077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.730827 7279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.731208 7077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:02.731295 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
uuid: "aeaf33431e9d42a19945e33ba8c02055"
format_stamp: "Formatted at 2026-09-27 00:53:02 on dist-test-slave-9fh2"
server_key: "b5e9ce0b0936dde78c69752a8dd78694"
server_key_iv: "b4afdba597699e516e05e37b248560c0"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.731396 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.738103 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:02.740553 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:02.746770 7077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46263
I20260927 00:53:02.747586 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:53:02.748090 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46263 every 8 connection(s)
I20260927 00:53:02.749568 7332 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:53:02.750859 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Bootstrap starting.
I20260927 00:53:02.761010 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:53:02.761286 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Bootstrap complete.
I20260927 00:53:02.761785 7332 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } }
I20260927 00:53:02.761900 7332 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeaf33431e9d42a19945e33ba8c02055, State: Initialized, Role: LEARNER
I20260927 00:53:02.761999 7332 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } }
I20260927 00:53:02.762313 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:02.762423 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:02.762449 7332 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:02.764989 7349 catalog_manager.cc:1259] Loaded cluster ID: d0d5a3fa083c4f2a9530f06c9f10ba31
I20260927 00:53:02.765049 7349 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: loading cluster ID for follower catalog manager: success
W20260927 00:53:02.765245 7257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:02.765698 7257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:02.765777 7257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:53:02.767012 7349 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: acquiring CA information for follower catalog manager: success
I20260927 00:53:02.768179 7349 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:53:02.770737 7257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:02.770844 7257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:02.770941 7257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:02.777303 7257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38639
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:02.778741 7257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:02.780088 7257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:02.792569 7354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.792695 7352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:02.792718 7351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:02.793011 7257 server_base.cc:1034] running on GCE node
I20260927 00:53:02.793780 7257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:02.794565 7257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:02.795758 7257 hybrid_clock.cc:648] HybridClock initialized: now 1790470382795730 us; error 55 us; skew 500 ppm
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:02.798671 7257 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:53:02.799681 7257 webserver.cc:460] Webserver started at http://127.0.98.1:38639/ using document root <none> and password file <none>
I20260927 00:53:02.800284 7257 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:02.800357 7257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:02.804980 7257 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:53:02.807765 7361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.809412 7257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:02.809543 7257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "d54f459a136143f59664ed20a1f2ea82"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "d757fb65f5d7db251f372603055be9b5"
server_key_iv: "d72a684e73ec1a087533094d602d92ce"
server_key_version: "encryptionkey@0"
I20260927 00:53:02.810205 7257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:02.831283 7257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:02.834789 7257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:02.834960 7257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:02.835580 7257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:02.836501 7257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:02.836552 7257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.836606 7257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:02.836633 7257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:02.846745 7257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36749
I20260927 00:53:02.846822 7474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36749 every 8 connection(s)
I20260927 00:53:02.847892 7257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:02.851562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 7257
I20260927 00:53:02.851758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6701
I20260927 00:53:02.860426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41683
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42173
--webserver_port=39637
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:53:02 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470382, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:02.868320 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.850120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48651 (local address 127.0.98.61:33443)
0927 00:53:02.850317 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.850322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.851439 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 00:53:02.853139 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.853143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.853209 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.853341 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.855298 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.856028 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.858014 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.858250 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.858358 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.862439 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.862443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.862447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.862480 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.864209 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.865579 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.865584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.865586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.865702 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.866204 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.866207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.866209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.866488 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.866581 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.867945 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.868103 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:53:02.869324 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.851861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.98.1:49989)
0927 00:53:02.853299 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.853313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.853432 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.853783 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.853787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.854317 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.855091 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.855122 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.857498 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.857502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.858421 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.858433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.859091 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.859095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.859250 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.859712 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.859726 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.861713 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.865700 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.865704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.865706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.865985 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.866317 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.866320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.866323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.866428 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.866961 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.866964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.867397 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.867650 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.868124 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":34,"spinlock_wait_cycles":2944,"thread_start_us":107,"threads_started":1}
I20260927 00:53:02.870072 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.850259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50625 (local address 127.0.98.62:42173)
0927 00:53:02.850443 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.850448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.859936 (+ 9488us) server_negotiation.cc:407] Connection header received
0927 00:53:02.860124 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.860128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.860202 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.860298 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.861021 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.861747 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.863050 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.863410 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.863514 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.866418 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.866422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.866425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.866458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.868149 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.868621 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.868624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.868626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.868717 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.869054 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.869057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.869059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.869308 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.869382 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.869749 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.869910 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:53:02.870097 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.852268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49989 (local address 127.0.98.60:46263)
0927 00:53:02.852577 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:02.852582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:02.853506 (+ 924us) server_negotiation.cc:407] Connection header received
0927 00:53:02.853559 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:02.853561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:02.853617 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:02.853709 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:02.856223 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.857348 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.858554 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.858952 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.859063 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.861956 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:02.861961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:02.861964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:02.861995 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:02.865517 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.866105 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.866108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.866110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.866200 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:02.866533 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:02.866537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:02.866538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:02.866816 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:02.866913 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:02.869773 (+ 2860us) server_negotiation.cc:299] Negotiation successful
0927 00:53:02.869922 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
I20260927 00:53:02.870772 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.850637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.98.1:48651)
0927 00:53:02.851233 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.851299 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.852882 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.853599 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.853609 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.854316 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.855088 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.855109 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.856190 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.856193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.857895 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.857906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.858579 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.858583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.858847 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.859617 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.859643 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.862198 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.864887 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.864896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.864923 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.865407 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.865824 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.865827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.865829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.866085 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.866638 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.866648 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.867399 (+ 751us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.867648 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.870565 (+ 2917us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1100,"spinlock_wait_cycles":1792,"thread_start_us":176,"threads_started":1}
I20260927 00:53:02.871568 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:02.850636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:50625)
0927 00:53:02.851231 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:02.859859 (+ 8628us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:02.859977 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:02.860341 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:02.860344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:02.860723 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:02.860872 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.860880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.861957 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.861960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.862924 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:02.862938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:02.863704 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:02.863707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:02.863781 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:02.864214 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:02.864229 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:02.866236 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:02.868269 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.868272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.868274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.868501 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.868829 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:02.868832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:02.868834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:02.868951 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:02.869457 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:02.869460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:02.869561 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:53:02.870813 (+ 1252us) client_negotiation.cc:240] Negotiation successful
0927 00:53:02.871009 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":111,"threads_started":1}
I20260927 00:53:02.872579 7477 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33443
I20260927 00:53:02.872643 7475 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46263
I20260927 00:53:02.872908 7475 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:02.872946 7476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 00:53:02.872912 7477 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:02.873046 7476 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:02.873766 7477 heartbeater.cc:511] Master 127.0.98.61:33443 requested a full tablet report, sending...
I20260927 00:53:02.873766 7475 heartbeater.cc:511] Master 127.0.98.60:46263 requested a full tablet report, sending...
I20260927 00:53:02.874504 7476 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 00:53:02.875777 7288 ts_manager.cc:194] Registered new tserver with Master: d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749)
I20260927 00:53:02.875890 7113 ts_manager.cc:194] Registered new tserver with Master: d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749)
I20260927 00:53:02.876658 6505 ts_manager.cc:194] Re-registered known tserver with Master: d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749)
I20260927 00:53:02.877354 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50625
I20260927 00:53:02.951402 6964 heartbeater.cc:503] Master 127.0.98.62:42173 was elected leader, sending a full tablet report...
W20260927 00:53:02.985132 7483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:02.985455 7483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:02.985504 7483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:02.989737 7483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:02.989818 7483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:02.989892 7483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:02.995734 7483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39637
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:02.996948 7483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:02.998183 7483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:03.009106 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:03.009158 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:03.009106 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:03.010366 7483 server_base.cc:1034] running on GCE node
I20260927 00:53:03.010903 7483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:03.011685 7483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:03.015123 7483 hybrid_clock.cc:648] HybridClock initialized: now 1790470383014997 us; error 86 us; skew 500 ppm
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:03.018913 7483 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:53:03.020282 7483 webserver.cc:460] Webserver started at http://127.0.98.2:39637/ using document root <none> and password file <none>
I20260927 00:53:03.021020 7483 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:03.021118 7483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:03.026341 7483 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 00:53:03.029517 7498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:03.031368 7483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:53:03.031543 7483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "8d021429bbf54df0bf1788a6017f91fd"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "46410834e548c160b6f4db01c0062f10"
server_key_iv: "59245b511d612003245f949e4f5b2b4a"
server_key_version: "encryptionkey@0"
I20260927 00:53:03.032370 7483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:03.056546 7483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:03.060464 7483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:03.060681 7483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:03.061478 7483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:03.062592 7483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:03.062673 7483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:03.062721 7483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:03.062753 7483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:03.073593 7483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41683
I20260927 00:53:03.073648 7611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41683 every 8 connection(s)
I20260927 00:53:03.074870 7483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:03.079963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 7483
I20260927 00:53:03.080178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6835
I20260927 00:53:03.082422 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.077449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.0.1:58164)
0927 00:53:03.077772 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.077788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.077886 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.078394 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.078397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.078421 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:03.078559 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.078576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.079715 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.079719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.080860 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.080873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.081959 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.081962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.082055 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.082081 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.082148 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.082250 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260927 00:53:03.082422 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.077764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.98.60:46263)
0927 00:53:03.077902 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.077907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.077928 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:53:03.078011 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.078014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.078098 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.078229 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:03.078725 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.079565 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.081312 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.081820 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.081938 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.082127 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.082164 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.082250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:53:03.083254 7306 raft_consensus.cc:1203] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 00:53:03.083727 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9aacfc0faad4d6b81ae8968aeaadc07. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:03.083894 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:53:03.087176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39593
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42173
--webserver_port=46495
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:53:03.097206 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.077123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42453 (local address 127.0.98.62:42173)
0927 00:53:03.077258 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.077264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.078592 (+ 1328us) server_negotiation.cc:407] Connection header received
0927 00:53:03.079857 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.079861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.079916 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.079985 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.085826 (+ 5841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.086588 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.087857 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.088183 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.088256 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.092417 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.092423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.092426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.092467 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.094367 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.094954 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.094957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.094960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.095048 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095551 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.095554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.095556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.095814 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.095903 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.096830 (+ 927us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.097004 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:53:03.097206 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.077485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60061 (local address 127.0.98.61:33443)
0927 00:53:03.077609 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.077614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.079442 (+ 1828us) server_negotiation.cc:407] Connection header received
0927 00:53:03.080063 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.080067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.080122 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.080173 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.085436 (+ 5263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.086247 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.087703 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.088014 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.088087 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.091756 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.091760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.091763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.091797 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.094025 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.094954 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.094957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.094960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.095048 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095550 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.095554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.095556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.095823 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.095903 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.096830 (+ 927us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.097004 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:53:03.098553 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.077783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.98.2:60061)
0927 00:53:03.079329 (+ 1546us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.079346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.079930 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.081896 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.081906 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.082454 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:03.085264 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.085284 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.086426 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.086429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.087555 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.087568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.088326 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.088330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.088436 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.089023 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:03.089047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:03.091487 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:03.094186 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.094195 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.094223 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.094835 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.095184 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.095188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.095439 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.095976 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:03.095984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:03.096504 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.097986 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.098141 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"mutex_wait_us":1188,"spinlock_wait_cycles":56960,"thread_start_us":175,"threads_started":1}
I20260927 00:53:03.098836 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.079002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.98.2:55721)
0927 00:53:03.082820 (+ 3818us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.082836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.082979 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.083274 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.083277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.083488 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:03.084351 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.084376 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.085462 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.085466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.087069 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.087078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.087892 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.087896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.088177 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.089186 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:03.089199 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:03.092459 (+ 3260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:03.094566 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.094570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.094572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.094935 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.095288 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.095293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.095444 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.095984 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:03.095987 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:03.096505 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.096761 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.097196 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3727,"mutex_wait_us":82,"spinlock_wait_cycles":2944,"thread_start_us":102,"threads_started":1}
I20260927 00:53:03.098553 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.077782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:42453)
0927 00:53:03.078439 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.078493 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.079630 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.084976 (+ 5346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.084979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.085508 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:03.085633 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.085639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.086734 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.086737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.087734 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.087743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.088453 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.088456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.088563 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.089095 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:03.089112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:03.092190 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:03.094485 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.094488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.094490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.094854 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.095181 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.095186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.095445 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.095976 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:03.095984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:03.096504 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.097405 (+ 901us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.097604 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":27,"spinlock_wait_cycles":31360,"thread_start_us":964,"threads_started":1}
I20260927 00:53:03.099555 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.079421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55721 (local address 127.0.98.60:46263)
0927 00:53:03.079690 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.079695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.083046 (+ 3351us) server_negotiation.cc:407] Connection header received
0927 00:53:03.083076 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.083078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.083121 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.083224 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.084525 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.085287 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.087238 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.087569 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.087663 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.092683 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.092688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.092691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.092723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.094421 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095057 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.095060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.095062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.095179 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.095559 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.095562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.095564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.095830 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.095917 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.099234 (+ 3317us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.099379 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260927 00:53:03.100587 7612 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46263
I20260927 00:53:03.100638 7613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 00:53:03.100888 7613 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:03.100886 7612 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:03.101596 7613 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 00:53:03.101595 7612 heartbeater.cc:511] Master 127.0.98.60:46263 requested a full tablet report, sending...
I20260927 00:53:03.102295 7614 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33443
I20260927 00:53:03.102383 7614 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:03.102569 7614 heartbeater.cc:511] Master 127.0.98.61:33443 requested a full tablet report, sending...
I20260927 00:53:03.103297 7288 ts_manager.cc:194] Registered new tserver with Master: 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2:41683)
I20260927 00:53:03.103556 6505 ts_manager.cc:194] Re-registered known tserver with Master: 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2:41683)
I20260927 00:53:03.103946 7113 ts_manager.cc:194] Registered new tserver with Master: 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2:41683)
I20260927 00:53:03.104331 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42453
W20260927 00:53:03.204972 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:03.205276 7620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:03.205338 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:03.209486 7620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:03.209563 7620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:03.209650 7620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:03.215107 7620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46495
--tserver_master_addrs=127.0.98.62:42173,127.0.98.61:33443,127.0.98.60:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:03.216271 7620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:03.217312 7620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:03.227978 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:03.228044 7620 server_base.cc:1034] running on GCE node
W20260927 00:53:03.227993 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:03.228005 7628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:03.228943 7620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:03.229624 7620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:03.230803 7620 hybrid_clock.cc:648] HybridClock initialized: now 1790470383230771 us; error 49 us; skew 500 ppm
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:03.233973 7620 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:53:03.235105 7620 webserver.cc:460] Webserver started at http://127.0.98.3:46495/ using document root <none> and password file <none>
I20260927 00:53:03.235760 7620 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:03.235850 7620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:03.240386 7620 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:53:03.243028 7635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:03.244679 7620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:53:03.244820 7620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "23d56d004b164a5c892209495071c41b"
format_stamp: "Formatted at 2026-09-27 00:53:01 on dist-test-slave-9fh2"
server_key: "fba098c652c6d0187f4a24d5b8a22783"
server_key_iv: "d23b0529e197cb99d6c6b872e3f9ddd5"
server_key_version: "encryptionkey@0"
I20260927 00:53:03.245409 7620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:03.260715 7620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:03.265179 7620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:03.265415 7620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:03.266194 7620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:03.267333 7620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:03.267387 7620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:03.267438 7620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:03.267472 7620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:03.278343 7620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39593
I20260927 00:53:03.278406 7748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39593 every 8 connection(s)
I20260927 00:53:03.279742 7620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:03.285327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 7620
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:53:03 dist-test-slave-9fh2 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:03.300844 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.284014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34867 (local address 127.0.98.60:46263)
0927 00:53:03.284151 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.284156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.284477 (+ 321us) server_negotiation.cc:407] Connection header received
0927 00:53:03.285642 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.285646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.285718 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.285775 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.287366 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.288081 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.290503 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.290886 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.290987 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.295871 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.295876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.295879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.295917 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.297737 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.298483 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.298488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.298490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.298588 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.299033 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.299037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.299039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.299303 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.299390 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.300443 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.300623 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:53:03.301963 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.283856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.98.3:34867)
0927 00:53:03.284348 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.284364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.284779 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.285968 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.285971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.286269 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:03.287107 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.287138 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.288252 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.288255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.290299 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.290319 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.291206 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.291209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.291358 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.292104 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:03.292120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:03.294772 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:03.297870 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.297879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.297906 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.298366 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.298688 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.298692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.298694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.298911 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.299444 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:03.299452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:03.299895 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.300153 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.301143 (+ 990us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":981,"spinlock_wait_cycles":66048,"thread_start_us":98,"threads_started":1}
I20260927 00:53:03.302251 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.282632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:42333)
0927 00:53:03.283408 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.283484 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.284747 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.285365 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.285368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.286063 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:03.287110 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.287135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.288391 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.288394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.290299 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.290319 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.291192 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.291197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.291370 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.292875 (+ 1505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:03.292891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:03.295114 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:03.298297 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.298301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.298303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.298597 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.298976 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.298979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.298981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.299074 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.300071 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:03.300074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:03.300193 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.300289 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.300685 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":624,"spinlock_wait_cycles":46080,"thread_start_us":774,"threads_started":1}
I20260927 00:53:03.302834 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.282210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42333 (local address 127.0.98.62:42173)
0927 00:53:03.282385 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.282390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.283656 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 00:53:03.285023 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.285029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.285104 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.285163 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.287350 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.288225 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.290524 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.290888 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.290987 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.295403 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.295408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.295411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.295444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.298137 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.298756 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.298760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.298762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.298869 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.299673 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.299676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.299678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.299924 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.300023 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.302513 (+ 2490us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.302673 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:53:03.303140 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.288817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.98.61:33443)
0927 00:53:03.290529 (+ 1712us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.290533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.290545 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:53:03.290574 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.290577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.290625 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.290710 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.293376 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.294313 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.295601 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.296466 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.296582 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.298376 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.298380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.298383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.298408 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.300072 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.300767 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.300770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.300772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.300855 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.301462 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.301466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.301468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.301717 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.301867 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.302832 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.302973 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":97,"threads_started":1}
I20260927 00:53:03.304224 7749 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46263
I20260927 00:53:03.304286 7750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 00:53:03.304507 7749 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:03.304507 7750 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:03.305145 7750 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 00:53:03.305526 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.282632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.98.3:53313)
0927 00:53:03.283733 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.283750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.284735 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.285355 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.285367 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.286063 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:03.287108 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.287135 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.288354 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.288357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.290274 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.290289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.291058 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.291063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.291290 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.291988 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:03.292007 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:03.295114 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:03.303015 (+ 7901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.303018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.303021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.303323 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.303650 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:03.303659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:03.303661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:03.303748 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:03.304999 (+ 1251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:03.305003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:03.305100 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.305205 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.305354 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":483,"spinlock_wait_cycles":75136,"thread_start_us":177,"threads_started":1}
I20260927 00:53:03.306363 6505 ts_manager.cc:194] Re-registered known tserver with Master: 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:03.306427 7749 heartbeater.cc:511] Master 127.0.98.60:46263 requested a full tablet report, sending...
I20260927 00:53:03.306278 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.282101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53313 (local address 127.0.98.61:33443)
0927 00:53:03.282272 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.282277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.283831 (+ 1554us) server_negotiation.cc:407] Connection header received
0927 00:53:03.284994 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.284997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.285060 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.285137 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.287341 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.288234 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.290447 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.290761 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.290852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.295403 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.295408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.295411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.295444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.300868 (+ 5424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.303460 (+ 2592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.303464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.303466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.303556 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.303845 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.303848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.303849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.304067 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.304165 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.305771 (+ 1606us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.306147 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 00:53:03.307039 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33443
I20260927 00:53:03.307135 7751 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:03.307281 6505 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42333
I20260927 00:53:03.307318 7751 heartbeater.cc:511] Master 127.0.98.61:33443 requested a full tablet report, sending...
I20260927 00:53:03.308017 7113 ts_manager.cc:194] Registered new tserver with Master: 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:03.308018 7288 ts_manager.cc:194] Registered new tserver with Master: 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:03.310920 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.292579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.0.98.62:42173)
0927 00:53:03.294096 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.294102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.296211 (+ 2109us) server_negotiation.cc:407] Connection header received
0927 00:53:03.296416 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.296419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.296498 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.296574 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.297694 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.298502 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.299640 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.299964 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.300041 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.305981 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.305985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.305988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.306019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.307789 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.309262 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.309266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.309268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.309364 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.309714 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.309719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.309721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.309978 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.310127 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.310533 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.310711 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":136,"threads_started":1}
I20260927 00:53:03.313198 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.289609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.98.60:46263)
0927 00:53:03.289736 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.289740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.293569 (+ 3829us) server_negotiation.cc:407] Connection header received
0927 00:53:03.293756 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.293759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.293814 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.293866 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.294827 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.295619 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.304880 (+ 9261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.305244 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.305359 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.309252 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.309255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.309257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.309281 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.311122 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.311654 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.311658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.311659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.311762 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.312057 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.312059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.312062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.312274 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.312395 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.312866 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.313045 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:53:03.314746 6505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:53:03.318751 6505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:03.322459 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: true } } }
I20260927 00:53:03.322659 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:03.326478 7768 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: attempting to promote NON_VOTER aeaf33431e9d42a19945e33ba8c02055 to VOTER
I20260927 00:53:03.326953 7768 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } }
I20260927 00:53:03.328558 7122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Refusing update from remote peer c9aacfc0faad4d6b81ae8968aeaadc07: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:53:03.329190 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:53:03.329931 7306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 LEARNER]: Refusing update from remote peer c9aacfc0faad4d6b81ae8968aeaadc07: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:53:03.330366 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:53:03.331271 7766 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.331864 7122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.333123 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.333283 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:03.333467 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.333588 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:03.334800 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.334947 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:03.335206 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.335322 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:03.336091 7306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.339331 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:03.339566 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:03.344403 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.337369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41683 (local address 127.0.0.1:48110)
0927 00:53:03.337501 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.337520 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.337624 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.339172 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.339176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.339194 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:03.339313 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.339320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.341403 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.341407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.342365 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.342379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.343775 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.343778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.343908 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.344070 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.344143 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.344242 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":102}
I20260927 00:53:03.344573 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.337676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.0.98.2:41683)
0927 00:53:03.338569 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.338583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.338622 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:53:03.338718 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.338724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.338983 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.339908 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:03.340169 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.341255 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.342567 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.343612 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.343962 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.344172 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.344243 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.344373 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":120,"threads_started":1}
I20260927 00:53:03.346539 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.338097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36749 (local address 127.0.0.1:44102)
0927 00:53:03.338732 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.338747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.338837 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.342055 (+ 3218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.342058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.342077 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:03.342187 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.342193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.343757 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.343761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.344890 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.344903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.346081 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.346085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.346186 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.346213 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.346299 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.346379 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260927 00:53:03.347957 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.338422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39593 (local address 127.0.0.1:42284)
0927 00:53:03.342316 (+ 3894us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.342330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.342416 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.343521 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.343523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.343539 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:03.343636 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.343641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.345262 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.345265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.346280 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.346288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.347437 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.347441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.347543 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.347589 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.347677 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.347780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3826,"thread_start_us":139,"threads_started":1}
I20260927 00:53:03.347980 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.339032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.0.98.3:39593)
0927 00:53:03.340585 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.340596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.342437 (+ 1841us) server_negotiation.cc:407] Connection header received
0927 00:53:03.342523 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.342529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.342672 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.343363 (+ 691us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:03.343762 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.345135 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.346396 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.347242 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.347356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.347518 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.347726 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.347832 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":155,"threads_started":1}
I20260927 00:53:03.348012 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.338311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.0.98.1:36749)
0927 00:53:03.341535 (+ 3224us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.341553 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.341578 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:53:03.341672 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.341678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.341916 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.342064 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:03.342349 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.343626 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.345055 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.345915 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.346058 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.346234 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.347720 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.347865 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3105,"thread_start_us":84,"threads_started":1}
I20260927 00:53:03.349009 7546 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca19bd403884c5ebf4510db884d283a (DEFAULT_TABLE table=test-table [id=cbbc84df12f4437eb21925db858f4166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:03.350592 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca19bd403884c5ebf4510db884d283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:03.351547 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca19bd403884c5ebf4510db884d283a (DEFAULT_TABLE table=test-table [id=cbbc84df12f4437eb21925db858f4166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:03.352896 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca19bd403884c5ebf4510db884d283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:03.354416 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca19bd403884c5ebf4510db884d283a (DEFAULT_TABLE table=test-table [id=cbbc84df12f4437eb21925db858f4166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:03.355640 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca19bd403884c5ebf4510db884d283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:03.361922 7782 tablet_bootstrap.cc:492] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Bootstrap starting.
I20260927 00:53:03.363713 7784 tablet_bootstrap.cc:492] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Bootstrap starting.
I20260927 00:53:03.363713 7783 tablet_bootstrap.cc:492] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Bootstrap starting.
I20260927 00:53:03.365731 7782 tablet_bootstrap.cc:654] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:03.366514 7782 log.cc:824] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:03.366568 7783 tablet_bootstrap.cc:654] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:03.366568 7784 tablet_bootstrap.cc:654] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:03.368033 7783 log.cc:824] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:03.368382 7784 log.cc:824] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:03.368654 7782 tablet_bootstrap.cc:492] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: No bootstrap required, opened a new log
I20260927 00:53:03.368889 7782 ts_tablet_manager.cc:1397] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:53:03.370501 7783 tablet_bootstrap.cc:492] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: No bootstrap required, opened a new log
I20260927 00:53:03.370745 7783 ts_tablet_manager.cc:1397] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:53:03.370746 7784 tablet_bootstrap.cc:492] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: No bootstrap required, opened a new log
I20260927 00:53:03.370968 7784 ts_tablet_manager.cc:1397] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260927 00:53:03.373651 7782 raft_consensus.cc:348] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.374567 7782 raft_consensus.cc:374] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:03.374639 7782 raft_consensus.cc:729] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d021429bbf54df0bf1788a6017f91fd, State: Initialized, Role: FOLLOWER
I20260927 00:53:03.375252 7782 consensus_queue.cc:260] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.375274 7783 raft_consensus.cc:348] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.375608 7783 raft_consensus.cc:374] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:03.375653 7783 raft_consensus.cc:729] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54f459a136143f59664ed20a1f2ea82, State: Initialized, Role: FOLLOWER
I20260927 00:53:03.376158 7783 consensus_queue.cc:260] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.376444 7782 ts_tablet_manager.cc:1428] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:53:03.376581 7784 raft_consensus.cc:348] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.376843 7784 raft_consensus.cc:374] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:03.376911 7784 raft_consensus.cc:729] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d56d004b164a5c892209495071c41b, State: Initialized, Role: FOLLOWER
I20260927 00:53:03.377295 7783 ts_tablet_manager.cc:1428] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:53:03.377403 7784 consensus_queue.cc:260] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.378377 7784 ts_tablet_manager.cc:1428] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:53:03.378852 7750 heartbeater.cc:503] Master 127.0.98.62:42173 was elected leader, sending a full tablet report...
I20260927 00:53:03.380808 7476 heartbeater.cc:503] Master 127.0.98.62:42173 was elected leader, sending a full tablet report...
I20260927 00:53:03.380955 7613 heartbeater.cc:503] Master 127.0.98.62:42173 was elected leader, sending a full tablet report...
W20260927 00:53:03.531422 7752 tablet.cc:2369] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:53:03.577445 7615 tablet.cc:2369] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:53:03.600389 7478 tablet.cc:2369] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:53:03.610270 7788 raft_consensus.cc:482] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:03.610520 7788 raft_consensus.cc:504] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.611784 7788 leader_election.cc:288] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749), 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:03.616333 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.612180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49601 (local address 127.0.98.1:36749)
0927 00:53:03.612338 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.612343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.612386 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:53:03.612556 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.612560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.612613 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.612684 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:03.612989 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.613769 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.615229 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.615724 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.615838 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.615918 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.616072 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.616161 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:53:03.616361 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.612036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36749 (local address 127.0.98.2:49601)
0927 00:53:03.612290 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.612306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.612445 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.612703 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.612706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.612723 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:03.612835 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.612841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.613896 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.613900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.615050 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.615061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.615859 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.615863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.615955 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.615978 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.616049 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.616162 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260927 00:53:03.616420 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.612401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41747 (local address 127.0.98.3:39593)
0927 00:53:03.612554 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.612559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.612865 (+ 306us) server_negotiation.cc:407] Connection header received
0927 00:53:03.612985 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.612988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.613040 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.613109 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:03.613437 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.614097 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.615426 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.615861 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.615963 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.616039 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.616206 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.616276 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:53:03.616496 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.612179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39593 (local address 127.0.98.2:41747)
0927 00:53:03.612737 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:03.612753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:03.612889 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:03.613144 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:03.613147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:03.613165 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:03.613326 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.613333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.614257 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.614260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.615227 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:03.615239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.615991 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.615995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:03.616093 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.616120 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:53:03.616185 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:53:03.616281 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":269,"thread_start_us":113,"threads_started":1}
I20260927 00:53:03.617590 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca19bd403884c5ebf4510db884d283a" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54f459a136143f59664ed20a1f2ea82" is_pre_election: true
I20260927 00:53:03.617749 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca19bd403884c5ebf4510db884d283a" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d56d004b164a5c892209495071c41b" is_pre_election: true
I20260927 00:53:03.617959 7429 raft_consensus.cc:2454] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d021429bbf54df0bf1788a6017f91fd in term 0.
I20260927 00:53:03.618140 7703 raft_consensus.cc:2454] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d021429bbf54df0bf1788a6017f91fd in term 0.
I20260927 00:53:03.618463 7500 leader_election.cc:302] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d021429bbf54df0bf1788a6017f91fd, d54f459a136143f59664ed20a1f2ea82; no voters:
I20260927 00:53:03.618781 7788 raft_consensus.cc:2781] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:03.618891 7788 raft_consensus.cc:482] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:03.618924 7788 raft_consensus.cc:3037] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:03.620285 7788 raft_consensus.cc:504] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.620724 7788 leader_election.cc:288] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 election: Requested vote from peers d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749), 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:03.621217 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca19bd403884c5ebf4510db884d283a" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d56d004b164a5c892209495071c41b"
I20260927 00:53:03.621217 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca19bd403884c5ebf4510db884d283a" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54f459a136143f59664ed20a1f2ea82"
I20260927 00:53:03.621390 7429 raft_consensus.cc:3037] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:03.621390 7703 raft_consensus.cc:3037] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:03.622857 7429 raft_consensus.cc:2454] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d021429bbf54df0bf1788a6017f91fd in term 1.
I20260927 00:53:03.622857 7703 raft_consensus.cc:2454] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d021429bbf54df0bf1788a6017f91fd in term 1.
I20260927 00:53:03.623236 7500 leader_election.cc:302] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d021429bbf54df0bf1788a6017f91fd, d54f459a136143f59664ed20a1f2ea82; no voters:
I20260927 00:53:03.623440 7788 raft_consensus.cc:2781] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:03.623641 7788 raft_consensus.cc:686] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 1 LEADER]: Becoming Leader. State: Replica: 8d021429bbf54df0bf1788a6017f91fd, State: Running, Role: LEADER
I20260927 00:53:03.624095 7788 consensus_queue.cc:237] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } }
I20260927 00:53:03.627935 6502 catalog_manager.cc:5495] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8d021429bbf54df0bf1788a6017f91fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } health_report { overall_health: UNKNOWN } } }
I20260927 00:53:03.712070 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.707623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.0.98.2:41683)
0927 00:53:03.707754 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.707757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.707775 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:53:03.707853 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.707855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.707912 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.708006 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:03.708397 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.708904 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.710252 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.710618 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.710717 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.711047 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:03.711617 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:03.711723 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.711838 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.711912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:53:03.718915 7429 raft_consensus.cc:1261] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Refusing update from remote peer 8d021429bbf54df0bf1788a6017f91fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:03.719009 7703 raft_consensus.cc:1261] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Refusing update from remote peer 8d021429bbf54df0bf1788a6017f91fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:03.719753 7788 consensus_queue.cc:1035] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:03.719982 7793 consensus_queue.cc:1035] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:03.729152 7796 mvcc.cc:205] Tried to move back new op lower bound from 7333766691704086528 to 7333766691327410176. Current Snapshot: MvccSnapshot[applied={T|T < 7333766691704086528}]
I20260927 00:53:03.732744 7798 mvcc.cc:205] Tried to move back new op lower bound from 7333766691704086528 to 7333766691327410176. Current Snapshot: MvccSnapshot[applied={T|T < 7333766691704086528}]
I20260927 00:53:03.732985 7797 mvcc.cc:205] Tried to move back new op lower bound from 7333766691704086528 to 7333766691327410176. Current Snapshot: MvccSnapshot[applied={T|T < 7333766691704086528}]
I20260927 00:53:03.740883 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:03.741139 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:03.747220 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9aacfc0faad4d6b81ae8968aeaadc07: Sending DeleteTablet for 3 replicas of tablet 6ca19bd403884c5ebf4510db884d283a
I20260927 00:53:03.748059 7546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca19bd403884c5ebf4510db884d283a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48110
I20260927 00:53:03.749147 7806 tablet_replica.cc:330] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: stopping tablet replica
I20260927 00:53:03.749361 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca19bd403884c5ebf4510db884d283a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44102
I20260927 00:53:03.749612 7806 raft_consensus.cc:2229] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:53:03.750017 7806 raft_consensus.cc:2258] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:03.751631 7806 ts_tablet_manager.cc:1905] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:03.751874 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca19bd403884c5ebf4510db884d283a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42284
I20260927 00:53:03.752838 7807 tablet_replica.cc:330] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: stopping tablet replica
I20260927 00:53:03.753708 7807 raft_consensus.cc:2229] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:03.754619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6473
I20260927 00:53:03.754489 7807 raft_consensus.cc:2258] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:03.756960 7806 ts_tablet_manager.cc:1918] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:03.757295 7808 tablet_replica.cc:330] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: stopping tablet replica
I20260927 00:53:03.757467 7807 ts_tablet_manager.cc:1905] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:03.757688 7806 log.cc:1196] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/6ca19bd403884c5ebf4510db884d283a
I20260927 00:53:03.757920 7808 raft_consensus.cc:2229] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:03.758332 7806 ts_tablet_manager.cc:1939] T 6ca19bd403884c5ebf4510db884d283a P 8d021429bbf54df0bf1788a6017f91fd: Deleting consensus metadata
I20260927 00:53:03.758756 7808 raft_consensus.cc:2258] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:03.760871 7807 ts_tablet_manager.cc:1918] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:03.760990 7807 log.cc:1196] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/6ca19bd403884c5ebf4510db884d283a
I20260927 00:53:03.761337 7807 ts_tablet_manager.cc:1939] T 6ca19bd403884c5ebf4510db884d283a P d54f459a136143f59664ed20a1f2ea82: Deleting consensus metadata
I20260927 00:53:03.763630 7808 ts_tablet_manager.cc:1905] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:03.768630 7808 ts_tablet_manager.cc:1918] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 00:53:03.768718 7362 connection.cc:535] server connection from 127.0.0.1:44102 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:53:03.768851 7808 log.cc:1196] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/6ca19bd403884c5ebf4510db884d283a
W20260927 00:53:03.769152 7500 connection.cc:535] server connection from 127.0.0.1:48110 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:53:03.769994 7808 ts_tablet_manager.cc:1939] T 6ca19bd403884c5ebf4510db884d283a P 23d56d004b164a5c892209495071c41b: Deleting consensus metadata
W20260927 00:53:03.771296 7639 connection.cc:441] server connection from 127.0.0.1:42284 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42284 (request call id 1) could send its response
I20260927 00:53:03.777684 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.753408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.0.98.61:33443)
0927 00:53:03.753576 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.753581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.760289 (+ 6708us) server_negotiation.cc:407] Connection header received
0927 00:53:03.760883 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.760887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.760943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.760995 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.763963 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.764784 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.767651 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.768137 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.768278 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.771553 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.771558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.771562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.771602 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.773499 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.774040 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.774044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.774046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.774126 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.774371 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.774374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.774376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.774597 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.774713 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.777385 (+ 2672us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.777509 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:53:03.784157 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.772788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.98.60:46263)
0927 00:53:03.772908 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.772912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.773714 (+ 802us) server_negotiation.cc:407] Connection header received
0927 00:53:03.775053 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.775057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.775113 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.775218 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.776247 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.777087 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.778244 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.778518 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.778588 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.780364 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.780368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.780371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.780395 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.782243 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.782686 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.782690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.782691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.782777 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.783088 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.783091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.783092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.783332 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.783435 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.783790 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.783902 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:53:03.784960 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.772395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.0.98.61:33443)
0927 00:53:03.772736 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.772740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.773514 (+ 774us) server_negotiation.cc:407] Connection header received
0927 00:53:03.773922 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.773925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.773982 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.774035 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.776204 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.777099 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.778402 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.778798 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.778879 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.780695 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.780699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.780702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.780736 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.783003 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.783542 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.783546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.783547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.783628 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.783921 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.783925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.783926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.784155 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.784267 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.784616 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.784760 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260927 00:53:03.786022 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:03.754027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.60:46263)
0927 00:53:03.754247 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:03.754252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:03.775607 (+ 21355us) server_negotiation.cc:407] Connection header received
0927 00:53:03.775654 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:03.775657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:03.775706 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:03.775776 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:03.776498 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.777227 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.780358 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:03.780673 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:03.780770 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:03.782305 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:03.782308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:03.782310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:03.782332 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:03.783917 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.784750 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.784753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.784755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.784828 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:03.785083 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:03.785087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:03.785089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:03.785318 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:03.785429 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:03.785782 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 00:53:03.785890 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:53:04.735513 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:04.734898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:60887)
0927 00:53:04.735289 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:04.735397 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":131,"threads_started":1}
W20260927 00:53:04.736070 7613 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:04.737373 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:04.736778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:43385)
0927 00:53:04.737162 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:04.737273 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":94,"threads_started":1}
W20260927 00:53:04.737677 7476 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:04.741253 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:04.740644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:51005)
0927 00:53:04.740974 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:04.741143 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
W20260927 00:53:04.741654 7750 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:05.248903 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9aacfc0faad4d6b81ae8968aeaadc07)
I20260927 00:53:05.249207 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } }
I20260927 00:53:05.250522 7820 leader_election.cc:288] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173), 3c84ad0ddc974750ab34976f75f05296 (127.0.98.61:33443)
I20260927 00:53:05.251384 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.250691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51814)
0927 00:53:05.251100 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.251228 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"spinlock_wait_cycles":23936,"thread_start_us":114,"threads_started":1}
W20260927 00:53:05.251629 7283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:05.252532 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9aacfc0faad4d6b81ae8968aeaadc07)
I20260927 00:53:05.252777 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } }
I20260927 00:53:05.252893 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.252522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51818)
0927 00:53:05.252718 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.252797 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 00:53:05.253046 7283 leader_election.cc:334] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:05.254215 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173), aeaf33431e9d42a19945e33ba8c02055 (127.0.98.60:46263)
I20260927 00:53:05.255017 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.254384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51822)
0927 00:53:05.254787 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.254883 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":86,"threads_started":1}
W20260927 00:53:05.255281 7100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:05.255944 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.250691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33443 (local address 127.0.0.1:34282)
0927 00:53:05.251163 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.251183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.251313 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.251723 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.251726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.251744 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.251882 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.251890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.252928 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.252933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.254503 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.254516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.255469 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.255473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.255578 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.255611 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.255708 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.255810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":29,"thread_start_us":114,"threads_started":1}
I20260927 00:53:05.256361 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.256100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51830)
0927 00:53:05.256248 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.256296 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 00:53:05.256488 7100 leader_election.cc:334] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:05.256697 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.250873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.0.98.61:33443)
0927 00:53:05.251193 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.251198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.251266 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:53:05.251486 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.251489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.251579 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.251692 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.252102 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.252769 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.254675 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.255338 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.255456 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.255580 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.256205 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.256311 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260927 00:53:05.257144 7122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aeaf33431e9d42a19945e33ba8c02055" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c84ad0ddc974750ab34976f75f05296" is_pre_election: true
I20260927 00:53:05.257356 7122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeaf33431e9d42a19945e33ba8c02055 in term 1.
I20260927 00:53:05.257755 7281 leader_election.cc:302] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c84ad0ddc974750ab34976f75f05296, aeaf33431e9d42a19945e33ba8c02055; no voters: c9aacfc0faad4d6b81ae8968aeaadc07
I20260927 00:53:05.258054 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:53:05.258105 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9aacfc0faad4d6b81ae8968aeaadc07)
I20260927 00:53:05.258140 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:53:05.258939 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.254643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.60:46263)
0927 00:53:05.254922 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.254931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.254945 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:53:05.255105 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.255113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.255197 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.255296 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.255777 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.256531 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.257862 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.258375 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.258479 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.258547 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.258724 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.258813 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":119,"threads_started":1}
I20260927 00:53:05.258953 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.254404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46263 (local address 127.0.0.1:58214)
0927 00:53:05.254866 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.254880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.254977 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.255328 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.255331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.255350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.255559 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.255565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.256640 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.256644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.257733 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.257743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.258499 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.258503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.258609 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.258633 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.258700 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.258798 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":18,"thread_start_us":82,"threads_started":1}
I20260927 00:53:05.259479 7306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c84ad0ddc974750ab34976f75f05296" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aeaf33431e9d42a19945e33ba8c02055" is_pre_election: true
I20260927 00:53:05.259584 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } }
I20260927 00:53:05.259884 7306 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c84ad0ddc974750ab34976f75f05296 in current term 2: Already voted for candidate aeaf33431e9d42a19945e33ba8c02055 in this term.
I20260927 00:53:05.260069 7820 leader_election.cc:288] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [CANDIDATE]: Term 2 election: Requested vote from peers c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173), 3c84ad0ddc974750ab34976f75f05296 (127.0.98.61:33443)
I20260927 00:53:05.260315 7098 leader_election.cc:302] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c84ad0ddc974750ab34976f75f05296; no voters: aeaf33431e9d42a19945e33ba8c02055, c9aacfc0faad4d6b81ae8968aeaadc07
I20260927 00:53:05.260466 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.260203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51836)
0927 00:53:05.260333 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.260405 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:53:05.260558 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:53:05.260563 7122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aeaf33431e9d42a19945e33ba8c02055" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c84ad0ddc974750ab34976f75f05296"
I20260927 00:53:05.261350 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.261149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51850)
0927 00:53:05.261255 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.261302 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:53:05.261453 7283 leader_election.cc:334] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
I20260927 00:53:05.261931 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:53:05.263532 7122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeaf33431e9d42a19945e33ba8c02055 in term 2.
I20260927 00:53:05.264175 7281 leader_election.cc:302] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c84ad0ddc974750ab34976f75f05296, aeaf33431e9d42a19945e33ba8c02055; no voters: c9aacfc0faad4d6b81ae8968aeaadc07
I20260927 00:53:05.264379 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:53:05.264644 7820 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [term 2 LEADER]: Becoming Leader. State: Replica: aeaf33431e9d42a19945e33ba8c02055, State: Running, Role: LEADER
I20260927 00:53:05.264988 7820 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } }
I20260927 00:53:05.266600 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeaf33431e9d42a19945e33ba8c02055. Latest consensus state: current_term: 2 leader_uuid: "aeaf33431e9d42a19945e33ba8c02055" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:05.266759 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:05.267122 7832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:05.269728 7832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:05.270164 7832 catalog_manager.cc:1259] Loaded cluster ID: d0d5a3fa083c4f2a9530f06c9f10ba31
I20260927 00:53:05.270213 7832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:05.270830 7832 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:05.271265 7832 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Loaded TSK: 0
I20260927 00:53:05.272118 7832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:53:05.308053 7288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:53:05.311492 7288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:05.315057 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.314735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51866)
0927 00:53:05.314878 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.314943 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:53:05.315126 7122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [term 2 FOLLOWER]: Refusing update from remote peer aeaf33431e9d42a19945e33ba8c02055: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 00:53:05.315682 7820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 00:53:05.315860 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.315668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.0.1:51876)
0927 00:53:05.315765 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.315813 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:53:05.316160 7283 consensus_peers.cc:487] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 -> Peer c9aacfc0faad4d6b81ae8968aeaadc07 (127.0.98.62:42173): Couldn't send request to peer c9aacfc0faad4d6b81ae8968aeaadc07. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:53:05.317477 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeaf33431e9d42a19945e33ba8c02055. Latest consensus state: current_term: 2 leader_uuid: "aeaf33431e9d42a19945e33ba8c02055" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:05.317642 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:05.319203 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "aeaf33431e9d42a19945e33ba8c02055" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:05.319386 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:05.320271 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "aeaf33431e9d42a19945e33ba8c02055" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c9aacfc0faad4d6b81ae8968aeaadc07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } peers { permanent_uuid: "3c84ad0ddc974750ab34976f75f05296" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33443 } attrs { promote: false } } peers { permanent_uuid: "aeaf33431e9d42a19945e33ba8c02055" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46263 } attrs { promote: false } } }
I20260927 00:53:05.320410 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c84ad0ddc974750ab34976f75f05296 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:53:05.330469 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.325687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41683 (local address 127.0.0.1:48114)
0927 00:53:05.325812 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.325830 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.325915 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.326574 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.326578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.326603 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.326743 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.326750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.327847 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.327850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.329091 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.329101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.329963 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.329967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.330050 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.330075 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.330142 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.330265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":38}
I20260927 00:53:05.330695 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.325906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39593 (local address 127.0.0.1:42290)
0927 00:53:05.326091 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.326105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.326176 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.326939 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.326942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.326958 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.327050 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.327056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.328394 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.328398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.329516 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.329524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.330331 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.330335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.330434 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.330453 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.330493 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.330557 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 00:53:05.330899 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.326148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36749 (local address 127.0.0.1:44118)
0927 00:53:05.326416 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.326429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.326508 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.327167 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.327170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.327191 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.327287 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.327293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.328265 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.328269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.329296 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.329305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.330191 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.330195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.330286 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.330313 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.330373 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.330453 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":66,"thread_start_us":143,"threads_started":1}
I20260927 00:53:05.331049 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.326107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.0.98.2:41683)
0927 00:53:05.326344 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.326348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.326361 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:53:05.326409 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.326412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.326460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.326547 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.326888 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.327717 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.329255 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.329849 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.329957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.330041 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.330792 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.330902 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260927 00:53:05.331569 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.326561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.0.98.1:36749)
0927 00:53:05.326847 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.326851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.326866 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:05.326914 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.326917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.326960 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.327037 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.327406 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.328152 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.329402 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.329963 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.330058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.330130 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.330518 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.331439 (+ 921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260927 00:53:05.331715 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.326171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.0.98.3:39593)
0927 00:53:05.326722 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.326726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.326741 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:05.326789 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.326792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.326836 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.326929 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.327182 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.328280 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.329634 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.330212 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.331002 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.331065 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.331113 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.331580 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":115,"threads_started":1}
I20260927 00:53:05.331735 7546 tablet_service.cc:1467] Processing CreateTablet for tablet 857e71a4f1ba4361bebeec776b1203f2 (DEFAULT_TABLE table=test-table [id=f9eabc86536142be9c1789da0e3764a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:05.332115 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857e71a4f1ba4361bebeec776b1203f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:05.332273 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 857e71a4f1ba4361bebeec776b1203f2 (DEFAULT_TABLE table=test-table [id=f9eabc86536142be9c1789da0e3764a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:05.332389 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 857e71a4f1ba4361bebeec776b1203f2 (DEFAULT_TABLE table=test-table [id=f9eabc86536142be9c1789da0e3764a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:05.332609 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857e71a4f1ba4361bebeec776b1203f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:05.332625 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857e71a4f1ba4361bebeec776b1203f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:05.334822 7843 tablet_bootstrap.cc:492] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Bootstrap starting.
I20260927 00:53:05.335741 7844 tablet_bootstrap.cc:492] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Bootstrap starting.
I20260927 00:53:05.336036 7843 tablet_bootstrap.cc:654] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:05.336704 7844 tablet_bootstrap.cc:654] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:05.337332 7845 tablet_bootstrap.cc:492] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82: Bootstrap starting.
I20260927 00:53:05.337464 7843 tablet_bootstrap.cc:492] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: No bootstrap required, opened a new log
I20260927 00:53:05.337616 7843 ts_tablet_manager.cc:1397] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:05.338075 7843 raft_consensus.cc:348] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.338152 7845 tablet_bootstrap.cc:654] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:05.338217 7843 raft_consensus.cc:374] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:05.338243 7843 raft_consensus.cc:729] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d021429bbf54df0bf1788a6017f91fd, State: Initialized, Role: FOLLOWER
I20260927 00:53:05.338389 7843 consensus_queue.cc:260] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.339337 7844 tablet_bootstrap.cc:492] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: No bootstrap required, opened a new log
I20260927 00:53:05.339332 7843 ts_tablet_manager.cc:1428] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:53:05.339432 7844 ts_tablet_manager.cc:1397] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:53:05.339720 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.339230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:54813)
0927 00:53:05.339565 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.339622 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260927 00:53:05.339882 7844 raft_consensus.cc:348] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.340011 7844 raft_consensus.cc:374] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:05.340052 7844 raft_consensus.cc:729] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d56d004b164a5c892209495071c41b, State: Initialized, Role: FOLLOWER
I20260927 00:53:05.340197 7844 consensus_queue.cc:260] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.340445 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.340253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:53511)
0927 00:53:05.340335 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.340385 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 00:53:05.340513 7844 ts_tablet_manager.cc:1428] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:05.341391 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.340953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:35859)
0927 00:53:05.341225 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.341282 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260927 00:53:05.342430 7749 heartbeater.cc:503] Master 127.0.98.60:46263 was elected leader, sending a full tablet report...
I20260927 00:53:05.342939 7612 heartbeater.cc:503] Master 127.0.98.60:46263 was elected leader, sending a full tablet report...
I20260927 00:53:05.343336 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.342600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:39099)
0927 00:53:05.343123 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.343223 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
I20260927 00:53:05.345549 7845 tablet_bootstrap.cc:492] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82: No bootstrap required, opened a new log
I20260927 00:53:05.345679 7845 ts_tablet_manager.cc:1397] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 00:53:05.346200 7845 raft_consensus.cc:348] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.346318 7845 raft_consensus.cc:374] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:05.346349 7845 raft_consensus.cc:729] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54f459a136143f59664ed20a1f2ea82, State: Initialized, Role: FOLLOWER
I20260927 00:53:05.346488 7845 consensus_queue.cc:260] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.346848 7845 ts_tablet_manager.cc:1428] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:53:05.348129 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.347298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:42483)
0927 00:53:05.347564 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.348014 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260927 00:53:05.348721 7475 heartbeater.cc:503] Master 127.0.98.60:46263 was elected leader, sending a full tablet report...
I20260927 00:53:05.348987 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.348581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:54273)
0927 00:53:05.348872 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.348915 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":120,"threads_started":1}
I20260927 00:53:05.354930 7854 raft_consensus.cc:482] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:05.355087 7854 raft_consensus.cc:504] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.356257 7854 leader_election.cc:288] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2:41683), 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:05.360637 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.356670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41725 (local address 127.0.98.2:41683)
0927 00:53:05.356807 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.356812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.356979 (+ 167us) server_negotiation.cc:407] Connection header received
0927 00:53:05.357064 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.357066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.357125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.357214 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.357673 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.358204 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.359592 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.359997 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.360086 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.360152 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.360398 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.360476 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:53:05.360656 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.356616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53365 (local address 127.0.98.3:39593)
0927 00:53:05.356726 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.356731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.356781 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:53:05.357053 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.357057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.357125 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.357214 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:05.357637 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.358195 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.359598 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.360055 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.360169 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.360250 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.360398 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.360489 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:53:05.360669 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.356537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39593 (local address 127.0.98.1:53365)
0927 00:53:05.356681 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.356698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.356856 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.357237 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.357240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.357266 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.357512 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.357519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.358308 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.358312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.359474 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.359485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.360185 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.360189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.360279 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.360301 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.360372 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.360492 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":137,"spinlock_wait_cycles":4864}
I20260927 00:53:05.360669 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.356537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41683 (local address 127.0.98.1:41725)
0927 00:53:05.356874 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.356887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:05.356972 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:05.357235 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:05.357238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:05.357264 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:05.357512 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.357519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.358329 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.358333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.359460 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:05.359472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.360149 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.360153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:05.360265 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.360294 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:53:05.360367 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:53:05.360489 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":256,"spinlock_wait_cycles":7936}
I20260927 00:53:05.361316 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "d54f459a136143f59664ed20a1f2ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d56d004b164a5c892209495071c41b" is_pre_election: true
I20260927 00:53:05.361513 7703 raft_consensus.cc:2454] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d54f459a136143f59664ed20a1f2ea82 in term 0.
I20260927 00:53:05.361483 7848 raft_consensus.cc:482] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:05.361483 7566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "d54f459a136143f59664ed20a1f2ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d021429bbf54df0bf1788a6017f91fd" is_pre_election: true
I20260927 00:53:05.361624 7848 raft_consensus.cc:504] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.361804 7566 raft_consensus.cc:2454] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d54f459a136143f59664ed20a1f2ea82 in term 0.
I20260927 00:53:05.362002 7848 leader_election.cc:288] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d56d004b164a5c892209495071c41b (127.0.98.3:39593), d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749)
I20260927 00:53:05.361974 7362 leader_election.cc:302] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d56d004b164a5c892209495071c41b, d54f459a136143f59664ed20a1f2ea82; no voters:
I20260927 00:53:05.362318 7854 raft_consensus.cc:2781] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:05.362416 7854 raft_consensus.cc:482] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:05.362457 7854 raft_consensus.cc:3037] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:05.362557 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54f459a136143f59664ed20a1f2ea82" is_pre_election: true
I20260927 00:53:05.362799 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d56d004b164a5c892209495071c41b" is_pre_election: true
I20260927 00:53:05.362967 7703 raft_consensus.cc:2454] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d021429bbf54df0bf1788a6017f91fd in term 0.
I20260927 00:53:05.363365 7499 leader_election.cc:302] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d56d004b164a5c892209495071c41b, 8d021429bbf54df0bf1788a6017f91fd; no voters:
I20260927 00:53:05.363624 7848 raft_consensus.cc:2781] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:05.363693 7848 raft_consensus.cc:482] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:05.363721 7848 raft_consensus.cc:3037] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:05.364003 7854 raft_consensus.cc:504] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.364238 7428 raft_consensus.cc:2379] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d021429bbf54df0bf1788a6017f91fd in current term 1: Already voted for candidate d54f459a136143f59664ed20a1f2ea82 in this term.
I20260927 00:53:05.364678 7566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "d54f459a136143f59664ed20a1f2ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d021429bbf54df0bf1788a6017f91fd"
I20260927 00:53:05.364920 7854 leader_election.cc:288] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [CANDIDATE]: Term 1 election: Requested vote from peers 8d021429bbf54df0bf1788a6017f91fd (127.0.98.2:41683), 23d56d004b164a5c892209495071c41b (127.0.98.3:39593)
I20260927 00:53:05.365103 7848 raft_consensus.cc:504] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.365221 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "d54f459a136143f59664ed20a1f2ea82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d56d004b164a5c892209495071c41b"
I20260927 00:53:05.365341 7703 raft_consensus.cc:3037] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:05.365373 7566 raft_consensus.cc:2379] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d54f459a136143f59664ed20a1f2ea82 in current term 1: Already voted for candidate 8d021429bbf54df0bf1788a6017f91fd in this term.
I20260927 00:53:05.365607 7848 leader_election.cc:288] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 election: Requested vote from peers 23d56d004b164a5c892209495071c41b (127.0.98.3:39593), d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749)
I20260927 00:53:05.365952 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d56d004b164a5c892209495071c41b"
I20260927 00:53:05.366468 7703 raft_consensus.cc:2454] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d54f459a136143f59664ed20a1f2ea82 in term 1.
I20260927 00:53:05.366799 7362 leader_election.cc:302] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23d56d004b164a5c892209495071c41b, d54f459a136143f59664ed20a1f2ea82; no voters: 8d021429bbf54df0bf1788a6017f91fd
I20260927 00:53:05.366921 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e71a4f1ba4361bebeec776b1203f2" candidate_uuid: "8d021429bbf54df0bf1788a6017f91fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54f459a136143f59664ed20a1f2ea82"
I20260927 00:53:05.367053 7854 raft_consensus.cc:2781] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:05.367393 7854 raft_consensus.cc:686] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 1 LEADER]: Becoming Leader. State: Replica: d54f459a136143f59664ed20a1f2ea82, State: Running, Role: LEADER
I20260927 00:53:05.367648 7854 consensus_queue.cc:237] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } }
I20260927 00:53:05.368247 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.367942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:59535)
0927 00:53:05.368082 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.368151 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:53:05.369135 7500 leader_election.cc:302] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d021429bbf54df0bf1788a6017f91fd; no voters: 23d56d004b164a5c892209495071c41b, d54f459a136143f59664ed20a1f2ea82
I20260927 00:53:05.369375 7848 raft_consensus.cc:2726] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 00:53:05.370277 7292 catalog_manager.cc:5495] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 reported cstate change: term changed from 0 to 1, leader changed from <none> to d54f459a136143f59664ed20a1f2ea82 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d54f459a136143f59664ed20a1f2ea82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d54f459a136143f59664ed20a1f2ea82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36749 } health_report { overall_health: HEALTHY } } }
I20260927 00:53:05.390540 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.387075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.0.98.1:36749)
0927 00:53:05.387254 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:05.387259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:05.387275 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:53:05.387353 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:05.387356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:05.387412 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:05.387515 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:05.387744 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.388301 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.389166 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:05.389377 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:05.389460 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:05.389692 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:05.390156 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:05.390250 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:05.390344 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:53:05.390404 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:53:05.394099 7703 raft_consensus.cc:1261] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Refusing update from remote peer d54f459a136143f59664ed20a1f2ea82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:05.394333 7566 raft_consensus.cc:1261] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Refusing update from remote peer d54f459a136143f59664ed20a1f2ea82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:05.394841 7857 consensus_queue.cc:1035] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d56d004b164a5c892209495071c41b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:05.395083 7854 consensus_queue.cc:1035] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d021429bbf54df0bf1788a6017f91fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:05.397418 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.397044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:49201)
0927 00:53:05.397226 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.397298 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 00:53:05.398227 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.397943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:60465)
0927 00:53:05.398061 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.398124 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 00:53:05.402240 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.400781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:35475)
0927 00:53:05.402057 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.402132 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1186}
I20260927 00:53:05.403045 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.402714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:54081)
0927 00:53:05.402875 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.402940 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 00:53:05.404249 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.403970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:49087)
0927 00:53:05.404120 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.404168 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 00:53:05.405151 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.404906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:56775)
0927 00:53:05.405019 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.405074 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:53:05.410899 7288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:05.411120 7288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:05.416028 7288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeaf33431e9d42a19945e33ba8c02055: Sending DeleteTablet for 3 replicas of tablet 857e71a4f1ba4361bebeec776b1203f2
I20260927 00:53:05.416858 7546 tablet_service.cc:1514] Processing DeleteTablet for tablet 857e71a4f1ba4361bebeec776b1203f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48114
I20260927 00:53:05.417021 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 857e71a4f1ba4361bebeec776b1203f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42290
I20260927 00:53:05.417062 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 857e71a4f1ba4361bebeec776b1203f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44118
I20260927 00:53:05.417459 7869 tablet_replica.cc:330] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: stopping tablet replica
I20260927 00:53:05.417610 7869 raft_consensus.cc:2229] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:05.417773 7869 raft_consensus.cc:2258] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:05.417999 7870 tablet_replica.cc:330] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82: stopping tablet replica
I20260927 00:53:05.418179 7870 raft_consensus.cc:2229] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:53:05.418442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 7257
I20260927 00:53:05.418676 7870 raft_consensus.cc:2258] T 857e71a4f1ba4361bebeec776b1203f2 P d54f459a136143f59664ed20a1f2ea82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:05.419082 7871 tablet_replica.cc:330] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: stopping tablet replica
I20260927 00:53:05.419343 7871 raft_consensus.cc:2229] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:05.419551 7871 raft_consensus.cc:2258] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:05.421054 7871 ts_tablet_manager.cc:1905] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:05.421212 7869 ts_tablet_manager.cc:1905] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:05.424335 7871 ts_tablet_manager.cc:1918] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:05.424335 7869 ts_tablet_manager.cc:1918] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:05.424451 7869 log.cc:1196] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/857e71a4f1ba4361bebeec776b1203f2
I20260927 00:53:05.424510 7871 log.cc:1196] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/857e71a4f1ba4361bebeec776b1203f2
I20260927 00:53:05.424775 7869 ts_tablet_manager.cc:1939] T 857e71a4f1ba4361bebeec776b1203f2 P 8d021429bbf54df0bf1788a6017f91fd: Deleting consensus metadata
I20260927 00:53:05.424803 7871 ts_tablet_manager.cc:1939] T 857e71a4f1ba4361bebeec776b1203f2 P 23d56d004b164a5c892209495071c41b: Deleting consensus metadata
I20260927 00:53:05.425788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 7483
I20260927 00:53:05.426630 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.425868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36749 (local address 127.0.0.1:44132)
0927 00:53:05.426481 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.426538 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520}
W20260927 00:53:05.427742 7281 catalog_manager.cc:4584] TS d54f459a136143f59664ed20a1f2ea82 (127.0.98.1:36749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 857e71a4f1ba4361bebeec776b1203f2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36749: connect: Connection refused (error 111)
I20260927 00:53:05.427975 7280 catalog_manager.cc:4841] TS 23d56d004b164a5c892209495071c41b (127.0.98.3:39593): tablet 857e71a4f1ba4361bebeec776b1203f2 (table test-table [id=f9eabc86536142be9c1789da0e3764a3]) successfully deleted
I20260927 00:53:05.434250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 7620
I20260927 00:53:05.434387 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:05.434065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41683 (local address 127.0.0.1:48128)
0927 00:53:05.434225 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:05.434301 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:53:05.440615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 6969
I20260927 00:53:05.447414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 7077
2026-09-27T00:53:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5652 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:53:05 dist-test-slave-9fh2 krb5kdc[7874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:53:05 dist-test-slave-9fh2 krb5kdc[7874](info): set up 2 sockets
Sep 27 00:53:05 dist-test-slave-9fh2 krb5kdc[7874](info): commencing operation
krb5kdc: starting...
W20260927 00:53:06.420202 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:53:06 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:53:06.550436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38563
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:06.660319 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:06.660606 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:06.660665 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:06.664692 7887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:06.664759 7887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:53:06.664783 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:06.664805 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:06.664832 7887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:06.664860 7887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:06.670212 7887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38563
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:06.671366 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:06.672364 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:06.681495 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:06.681439 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:06.681561 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:06.681828 7887 server_base.cc:1034] running on GCE node
I20260927 00:53:06.682706 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:06.683893 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:06.685067 7887 hybrid_clock.cc:648] HybridClock initialized: now 1790470386685047 us; error 42 us; skew 500 ppm
Sep 27 00:53:06 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:06.692961 7887 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:53:06.694005 7887 webserver.cc:460] Webserver started at http://127.0.98.62:45007/ using document root <none> and password file <none>
I20260927 00:53:06.694566 7887 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:06.694629 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:06.694831 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:06.696926 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "62bb933a66c044c582ff3b61c6678a00"
format_stamp: "Formatted at 2026-09-27 00:53:06 on dist-test-slave-9fh2"
server_key: "e9604f0b6510bd12a8c802a83c0d8c3a"
server_key_iv: "33882e4b5efd24d374b09de2c52ea589"
server_key_version: "encryptionkey@0"
I20260927 00:53:06.697507 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "62bb933a66c044c582ff3b61c6678a00"
format_stamp: "Formatted at 2026-09-27 00:53:06 on dist-test-slave-9fh2"
server_key: "e9604f0b6510bd12a8c802a83c0d8c3a"
server_key_iv: "33882e4b5efd24d374b09de2c52ea589"
server_key_version: "encryptionkey@0"
I20260927 00:53:06.701648 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:53:06.704388 7902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:06.705871 7887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:06.705994 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "62bb933a66c044c582ff3b61c6678a00"
format_stamp: "Formatted at 2026-09-27 00:53:06 on dist-test-slave-9fh2"
server_key: "e9604f0b6510bd12a8c802a83c0d8c3a"
server_key_iv: "33882e4b5efd24d374b09de2c52ea589"
server_key_version: "encryptionkey@0"
I20260927 00:53:06.706110 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:06.717075 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:06.720355 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:06.720523 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:06.728854 7887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38563
I20260927 00:53:06.729017 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38563 every 8 connection(s)
I20260927 00:53:06.729822 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:06.733724 7965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:06.738360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 7887
I20260927 00:53:06.738507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:53:06.738835 392 external_mini_cluster.cc:1428] Setting key c34a65214f3a973882e228821627a610
I20260927 00:53:06.742585 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Bootstrap starting.
I20260927 00:53:06.745644 7965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:06.746692 7965 log.cc:824] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:53:06 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:06.749547 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: No bootstrap required, opened a new log
I20260927 00:53:06.754164 7965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bb933a66c044c582ff3b61c6678a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38563 } }
I20260927 00:53:06.754472 7965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:06.754520 7965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bb933a66c044c582ff3b61c6678a00, State: Initialized, Role: FOLLOWER
I20260927 00:53:06.755038 7965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bb933a66c044c582ff3b61c6678a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38563 } }
I20260927 00:53:06.755251 7965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:06.755309 7965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:06.755391 7965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:06.756122 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:06.740542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.0.98.62:38563)
0927 00:53:06.741161 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:06.741205 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:06.741428 (+ 223us) server_negotiation.cc:407] Connection header received
0927 00:53:06.742419 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:06.742440 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:06.742885 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:06.743270 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:06.744229 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:06.745286 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:06.746534 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:06.747125 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:06.747314 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:06.750459 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:06.750466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:06.750488 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:06.750535 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:06.752619 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:06.753175 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:06.753181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:06.753187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:06.753349 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:06.753621 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:06.753624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:06.753625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:06.754149 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:06.754348 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:06.754688 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 00:53:06.755133 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":169,"threads_started":1}
I20260927 00:53:06.756636 7965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bb933a66c044c582ff3b61c6678a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38563 } }
I20260927 00:53:06.757017 7965 leader_election.cc:302] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62bb933a66c044c582ff3b61c6678a00; no voters:
I20260927 00:53:06.757366 7965 leader_election.cc:288] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:53:06.757580 7970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:06.757977 7970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [term 1 LEADER]: Becoming Leader. State: Replica: 62bb933a66c044c582ff3b61c6678a00, State: Running, Role: LEADER
I20260927 00:53:06.758422 7970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bb933a66c044c582ff3b61c6678a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38563 } }
I20260927 00:53:06.758705 7965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:06.760409 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62bb933a66c044c582ff3b61c6678a00. Latest consensus state: current_term: 1 leader_uuid: "62bb933a66c044c582ff3b61c6678a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bb933a66c044c582ff3b61c6678a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38563 } } }
I20260927 00:53:06.760571 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:06.760886 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62bb933a66c044c582ff3b61c6678a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62bb933a66c044c582ff3b61c6678a00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38563 } } }
I20260927 00:53:06.760967 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00 [sys.catalog]: This master's current role is: LEADER
W20260927 00:53:06.764683 7985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:53:06.764806 7985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:53:06.764899 7984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:06.766052 7984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:06.772328 7984 catalog_manager.cc:1347] Generated new cluster ID: 06dc43aeeab349af8b9a592db298f2fa
I20260927 00:53:06.772420 7984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:06.790242 7984 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:53:06.791904 7984 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:06.802677 7984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Generated new TSK 0
I20260927 00:53:06.803781 7984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:53:06.843050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38563
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:06.971939 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:06.972452 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:06.972532 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:06.977207 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:06.977293 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:06.977339 7991 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:53:06.977375 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:06.983387 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:06.984814 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:06.986044 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:06.997628 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:06.997607 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:06.997721 7991 server_base.cc:1034] running on GCE node
W20260927 00:53:06.997604 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:06.998437 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:06.999333 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:07.000526 7991 hybrid_clock.cc:648] HybridClock initialized: now 1790470387000502 us; error 42 us; skew 500 ppm
Sep 27 00:53:06 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:07.004027 7991 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:53:07.005313 7991 webserver.cc:460] Webserver started at http://127.0.98.1:35835/ using document root <none> and password file <none>
I20260927 00:53:07.006033 7991 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:07.006109 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:07.006361 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:07.009090 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
format_stamp: "Formatted at 2026-09-27 00:53:06 on dist-test-slave-9fh2"
server_key: "46dfd66f7ebf65d0c0bef6a221e9cd88"
server_key_iv: "8d4685c67aac56eb2f00235f1e199ad9"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.009794 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
format_stamp: "Formatted at 2026-09-27 00:53:06 on dist-test-slave-9fh2"
server_key: "46dfd66f7ebf65d0c0bef6a221e9cd88"
server_key_iv: "8d4685c67aac56eb2f00235f1e199ad9"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.014794 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 00:53:07.018523 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.020399 7991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:53:07.020565 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
format_stamp: "Formatted at 2026-09-27 00:53:06 on dist-test-slave-9fh2"
server_key: "46dfd66f7ebf65d0c0bef6a221e9cd88"
server_key_iv: "8d4685c67aac56eb2f00235f1e199ad9"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.020735 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:07.042512 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:07.046433 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:07.046648 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:07.048514 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:07.048580 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.048633 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:07.048650 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:07.060976 7991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37601
I20260927 00:53:07.061115 8121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37601 every 8 connection(s)
I20260927 00:53:07.062330 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:07.064666 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.049596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42793 (local address 127.0.98.62:38563)
0927 00:53:07.049728 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.049732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.051136 (+ 1404us) server_negotiation.cc:407] Connection header received
0927 00:53:07.052525 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.052531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.052618 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.052765 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.054142 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.055031 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.056562 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.056975 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.057142 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.060452 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.060455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.060458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.060488 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.062206 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.062894 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.062898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.062900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.063011 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.063489 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.063492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.063494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.063709 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.063801 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.064312 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.064481 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:53:07.065433 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.050302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.98.1:42793)
0927 00:53:07.050954 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.051012 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.052213 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.052963 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.052973 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.053431 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:07.053971 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.053985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.055255 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.055258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.056437 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.056449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.057205 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.057209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.057391 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.057993 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:07.058011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:07.060160 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:07.062366 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.062371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.062388 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.062741 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.063228 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.063231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.063233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.063374 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.063817 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:07.063824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:07.064072 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.064238 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.064575 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":159,"threads_started":1}
I20260927 00:53:07.066849 8122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38563
I20260927 00:53:07.067046 8122 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:07.067761 8122 heartbeater.cc:511] Master 127.0.98.62:38563 requested a full tablet report, sending...
I20260927 00:53:07.069556 7919 ts_manager.cc:194] Registered new tserver with Master: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601)
I20260927 00:53:07.071089 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42793
I20260927 00:53:07.071365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 7991
I20260927 00:53:07.071506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:53:07.071870 392 external_mini_cluster.cc:1428] Setting key 6cf5fc4554954ffaea94dc880bc3e7a2
I20260927 00:53:07.077358 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.069144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52923 (local address 127.0.98.62:38563)
0927 00:53:07.069265 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.069269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.069445 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:53:07.069569 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.069571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.069618 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.069712 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.070233 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.071029 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.072258 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.072472 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.072633 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.074028 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.074032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.074036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.074071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.075636 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.076156 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.076159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.076160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.076235 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.076548 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.076552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.076553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.076734 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.076830 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.077144 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.077246 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:53:07.077407 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.069047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.98.1:52923)
0927 00:53:07.069354 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.069368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.069474 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.069741 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.069744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.069989 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:07.070119 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.070125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.071261 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.071265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.072141 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.072152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.072645 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.072650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.072743 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.073379 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:07.073393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:07.073879 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:07.075746 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.075750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.075752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.076018 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.076328 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.076331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.076334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.076432 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.076831 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:07.076834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:07.076930 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.077044 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.077192 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:53:07.102850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38563
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:07.229110 8131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:07.229538 8131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:07.229605 8131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:07.234086 8131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:07.234177 8131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:07.234220 8131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:53:07.234257 8131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:07.240345 8131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:07.241804 8131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:07.243065 8131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:07.254828 8136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:07.254983 8131 server_base.cc:1034] running on GCE node
W20260927 00:53:07.254828 8137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:07.254858 8139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:07.255981 8131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:07.256745 8131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:07.257929 8131 hybrid_clock.cc:648] HybridClock initialized: now 1790470387257910 us; error 44 us; skew 500 ppm
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470387, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:07.261447 8131 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:53:07.262495 8131 webserver.cc:460] Webserver started at http://127.0.98.2:39931/ using document root <none> and password file <none>
I20260927 00:53:07.263137 8131 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:07.263250 8131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:07.263470 8131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:07.265625 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "2eb226f457104f0a8cb5acb52362ebf3"
format_stamp: "Formatted at 2026-09-27 00:53:07 on dist-test-slave-9fh2"
server_key: "70e5ccebdbb0d8a996186e1091435b0d"
server_key_iv: "aba350f4ea57881d6a9a314992c67a6a"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.266214 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2eb226f457104f0a8cb5acb52362ebf3"
format_stamp: "Formatted at 2026-09-27 00:53:07 on dist-test-slave-9fh2"
server_key: "70e5ccebdbb0d8a996186e1091435b0d"
server_key_iv: "aba350f4ea57881d6a9a314992c67a6a"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.270605 8131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:53:07.273782 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.275554 8131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:07.275714 8131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "2eb226f457104f0a8cb5acb52362ebf3"
format_stamp: "Formatted at 2026-09-27 00:53:07 on dist-test-slave-9fh2"
server_key: "70e5ccebdbb0d8a996186e1091435b0d"
server_key_iv: "aba350f4ea57881d6a9a314992c67a6a"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.275832 8131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:07.297219 8131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:07.300441 8131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:07.300633 8131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:07.302315 8131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:07.302390 8131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.302446 8131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:07.302467 8131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.313642 8131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41793
I20260927 00:53:07.313923 8261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41793 every 8 connection(s)
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470387, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:07.314677 8131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:07.319330 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.303708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58391 (local address 127.0.98.62:38563)
0927 00:53:07.303845 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.303849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.305055 (+ 1206us) server_negotiation.cc:407] Connection header received
0927 00:53:07.306621 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.306625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.306698 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.306782 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.308869 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.309692 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.311483 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.311841 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.311969 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.315357 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.315361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.315365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.315397 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.316918 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.317500 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.317503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.317504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.317601 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.317939 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.317942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.317943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.318178 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.318259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.318894 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.319065 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:53:07.319962 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.304254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.98.2:58391)
0927 00:53:07.304824 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.304906 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.306254 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.307079 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.307090 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.307725 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:07.308597 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.308622 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.309933 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.309936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.311309 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.311324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.312042 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.312049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.312335 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.313038 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:07.313060 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:07.315101 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:07.317058 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.317063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.317077 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.317352 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.317691 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.317694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.317695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.317835 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.318270 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:07.318276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:07.318577 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.318755 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.319068 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260927 00:53:07.321244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 8131
I20260927 00:53:07.321378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:53:07.321746 392 external_mini_cluster.cc:1428] Setting key 5acfe6c1f19af283bc32443abb697127
I20260927 00:53:07.321861 8262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38563
I20260927 00:53:07.322019 8262 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:07.322609 8262 heartbeater.cc:511] Master 127.0.98.62:38563 requested a full tablet report, sending...
I20260927 00:53:07.323868 7914 ts_manager.cc:194] Registered new tserver with Master: 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793)
I20260927 00:53:07.324729 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58391
I20260927 00:53:07.333312 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.324120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.98.2:58711)
0927 00:53:07.325231 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.325250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.325395 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.325845 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.325848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.326128 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:07.326272 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.326281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.327358 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.327360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.327934 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.327943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.328477 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.328479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.328550 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.329039 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:07.329051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:07.329431 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:07.331467 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.331470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.331471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.331678 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.332055 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.332058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.332059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.332167 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.332768 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:07.332771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:07.332862 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.332954 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.333147 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:53:07.333391 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.324209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58711 (local address 127.0.98.62:38563)
0927 00:53:07.324335 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.324339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.325349 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 00:53:07.325508 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.325512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.325567 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.325723 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.326411 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.327228 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.328051 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.328360 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.328462 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.329609 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.329614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.329617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.329650 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.331344 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.331789 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.331793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.331795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.331899 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.332282 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.332286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.332288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.332652 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.332781 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.333047 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.333194 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:53:07.356372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38563
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:07.486923 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:07.487340 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:07.487397 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:07.491812 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:07.491897 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:07.491950 8271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:53:07.491994 8271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:07.497979 8271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:07.499382 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:07.500599 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:07.512086 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:07.512115 8279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:07.512101 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:07.512202 8271 server_base.cc:1034] running on GCE node
I20260927 00:53:07.512878 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:07.513701 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:07.514873 8271 hybrid_clock.cc:648] HybridClock initialized: now 1790470387514832 us; error 64 us; skew 500 ppm
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470387, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:07.518034 8271 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:53:07.519198 8271 webserver.cc:460] Webserver started at http://127.0.98.3:32781/ using document root <none> and password file <none>
I20260927 00:53:07.519908 8271 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:07.519989 8271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:07.520190 8271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:07.522675 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "21565270df714824b8ce5bd321af42bf"
format_stamp: "Formatted at 2026-09-27 00:53:07 on dist-test-slave-9fh2"
server_key: "4ca5d8c9c7fbb7b518050db2080d04e1"
server_key_iv: "ef546210f3ee3d41a52f3d630c1b6ace"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.523306 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21565270df714824b8ce5bd321af42bf"
format_stamp: "Formatted at 2026-09-27 00:53:07 on dist-test-slave-9fh2"
server_key: "4ca5d8c9c7fbb7b518050db2080d04e1"
server_key_iv: "ef546210f3ee3d41a52f3d630c1b6ace"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.527961 8271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:53:07.531006 8286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.532579 8271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:07.532720 8271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "21565270df714824b8ce5bd321af42bf"
format_stamp: "Formatted at 2026-09-27 00:53:07 on dist-test-slave-9fh2"
server_key: "4ca5d8c9c7fbb7b518050db2080d04e1"
server_key_iv: "ef546210f3ee3d41a52f3d630c1b6ace"
server_key_version: "encryptionkey@0"
I20260927 00:53:07.532837 8271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:07.556315 8271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:07.560032 8271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:07.560212 8271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:07.562042 8271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:07.562101 8271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:07.562146 8271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:07.562175 8271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470387, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:07.574010 8271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43189
I20260927 00:53:07.574079 8401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43189 every 8 connection(s)
I20260927 00:53:07.575325 8271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:07.577234 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.562913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39073 (local address 127.0.98.62:38563)
0927 00:53:07.563036 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.563040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.564458 (+ 1418us) server_negotiation.cc:407] Connection header received
0927 00:53:07.565742 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.565746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.565799 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.565876 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.567374 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.568051 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.569249 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.569470 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.569553 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.572960 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.572963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.572966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.572989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.574863 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.575517 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.575521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.575523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.575610 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.575997 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.576000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.576002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.576230 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.576336 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.576969 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.577087 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:53:07.578166 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.563671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.98.3:39073)
0927 00:53:07.564293 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.564347 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.565512 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.566095 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.566105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.566547 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:07.567239 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.567256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.568162 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.568166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.569134 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.569144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.569624 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.569628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.569785 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.570378 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:07.570396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:07.572729 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:07.575031 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.575039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.575056 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.575405 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.575720 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.575723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.575726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.575904 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.576342 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:07.576349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:07.576726 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.576900 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.577231 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":145,"threads_started":1}
I20260927 00:53:07.579634 8402 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38563
I20260927 00:53:07.579818 8402 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:07.580464 8402 heartbeater.cc:511] Master 127.0.98.62:38563 requested a full tablet report, sending...
I20260927 00:53:07.581528 7919 ts_manager.cc:194] Registered new tserver with Master: 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:07.582139 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39073
I20260927 00:53:07.585353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 8271
I20260927 00:53:07.585480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:53:07.585810 392 external_mini_cluster.cc:1428] Setting key 668ff2e3edd19d9f322f279822272ecb
I20260927 00:53:07.588140 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:53:07.589468 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.582407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59715 (local address 127.0.98.62:38563)
0927 00:53:07.582522 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.582526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.582539 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:53:07.582674 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.582676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.582713 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.582772 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.583247 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.583738 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.584577 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.584781 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.584863 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.586118 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.586123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.586126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.586164 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.587820 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.588264 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.588269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.588271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.588370 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.588653 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.588655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.588656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.588846 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.588920 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.589213 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.589324 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:53:07.589506 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.582132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.98.3:59715)
0927 00:53:07.582460 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.582478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.582573 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.582802 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.582805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.583016 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:07.583134 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.583141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.583857 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.583860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.584462 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.584470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.584885 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.584889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.584976 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.585526 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:07.585540 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:07.585967 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:07.587924 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.587928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.587930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.588153 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.588463 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:07.588466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:07.588468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:07.588560 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:07.588937 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:07.588940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:07.589041 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.589171 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.589316 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:53:07.614519 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:07.607419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.0.98.62:38563)
0927 00:53:07.607574 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.607578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.607592 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:53:07.607643 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.607646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.607691 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.607763 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.608550 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.609175 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.610135 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.610399 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.610505 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.614319 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57190: BlockingRecv error: recv got EOF from 127.0.0.1:57190 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:53:07 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:07.625725 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.616661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.0.98.62:38563)
0927 00:53:07.616821 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.616826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.616916 (+ 90us) server_negotiation.cc:407] Connection header received
0927 00:53:07.617101 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.617104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.617154 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.617226 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:07.617948 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.618508 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.619444 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.619707 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.619802 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.622452 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:07.622455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:07.622458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:07.622487 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:07.624024 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.624460 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.624464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.624466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.624539 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:07.624860 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:07.624864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:07.624866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:07.625076 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:07.625205 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.625423 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.625539 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:53:07.629441 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:53:07.632632 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:07.648523 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.642661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37601 (local address 127.0.0.1:39598)
0927 00:53:07.643698 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.643748 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.643963 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.644639 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.644644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.644665 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:07.644779 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.644783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.645780 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.645782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.647030 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.647039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.648004 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.648007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.648108 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.648137 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.648228 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.648367 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260927 00:53:07.648523 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.642884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.0.98.1:37601)
0927 00:53:07.643470 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.643476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.643837 (+ 361us) server_negotiation.cc:407] Connection header received
0927 00:53:07.644171 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.644176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.644369 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.644477 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:07.644887 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.645658 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.647192 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.647755 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.647820 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.647928 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.648279 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.648374 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":89,"threads_started":1}
I20260927 00:53:07.649787 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.642753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.0.98.3:43189)
0927 00:53:07.643221 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.643238 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.644275 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 00:53:07.644387 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.644393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.644654 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.644809 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:07.645115 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.646286 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.647869 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.648730 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.648841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.649036 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.649487 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.649620 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":128,"threads_started":1}
I20260927 00:53:07.649974 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.643855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.0.98.2:41793)
0927 00:53:07.645168 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.645185 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.645210 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:53:07.645307 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.645312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.645505 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.645675 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:07.645933 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.647045 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.648516 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.649302 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.649417 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.649587 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.649699 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.649813 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":128,"threads_started":1}
I20260927 00:53:07.649960 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.643617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41793 (local address 127.0.0.1:32998)
0927 00:53:07.644414 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.644429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.644521 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.645682 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.645686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.645706 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:07.645806 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.645812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.647237 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.647251 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.648358 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.648369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.649441 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.649445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.649545 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.649575 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.649667 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.649772 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":113,"threads_started":1}
I20260927 00:53:07.651105 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.642661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43189 (local address 127.0.0.1:55586)
0927 00:53:07.643971 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.643990 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.644116 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.644795 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.644798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.644818 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:07.644977 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.644986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.646456 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.646460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.647707 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.647717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.649207 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.649210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.649306 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.649327 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.650588 (+ 1261us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.650693 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1234,"mutex_wait_us":25,"thread_start_us":121,"threads_started":1}
I20260927 00:53:07.651880 8056 tablet_service.cc:1467] Processing CreateTablet for tablet d18c3d5210e041dcad11aada7ea0ed9a (DEFAULT_TABLE table=test-table [id=8b71d4f445c340a0a7228de27e8bb911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:07.652925 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18c3d5210e041dcad11aada7ea0ed9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:07.653349 8196 tablet_service.cc:1467] Processing CreateTablet for tablet d18c3d5210e041dcad11aada7ea0ed9a (DEFAULT_TABLE table=test-table [id=8b71d4f445c340a0a7228de27e8bb911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:07.654042 8336 tablet_service.cc:1467] Processing CreateTablet for tablet d18c3d5210e041dcad11aada7ea0ed9a (DEFAULT_TABLE table=test-table [id=8b71d4f445c340a0a7228de27e8bb911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:07.654628 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18c3d5210e041dcad11aada7ea0ed9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:07.655043 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18c3d5210e041dcad11aada7ea0ed9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:07.659137 8430 tablet_bootstrap.cc:492] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Bootstrap starting.
I20260927 00:53:07.662036 8430 tablet_bootstrap.cc:654] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:07.662744 8430 log.cc:824] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:07.664245 8433 tablet_bootstrap.cc:492] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Bootstrap starting.
I20260927 00:53:07.664682 8430 tablet_bootstrap.cc:492] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: No bootstrap required, opened a new log
I20260927 00:53:07.664855 8430 ts_tablet_manager.cc:1397] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:53:07.665704 8432 tablet_bootstrap.cc:492] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Bootstrap starting.
I20260927 00:53:07.667933 8432 tablet_bootstrap.cc:654] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:07.668780 8432 log.cc:824] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:07.669018 8433 tablet_bootstrap.cc:654] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:07.669425 8430 raft_consensus.cc:348] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.669668 8430 raft_consensus.cc:374] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:07.669719 8430 raft_consensus.cc:729] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0, State: Initialized, Role: FOLLOWER
I20260927 00:53:07.670014 8433 log.cc:824] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:07.670128 8430 consensus_queue.cc:260] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.670723 8122 heartbeater.cc:503] Master 127.0.98.62:38563 was elected leader, sending a full tablet report...
I20260927 00:53:07.671097 8430 ts_tablet_manager.cc:1428] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:53:07.671589 8432 tablet_bootstrap.cc:492] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: No bootstrap required, opened a new log
I20260927 00:53:07.671871 8432 ts_tablet_manager.cc:1397] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:53:07.672477 8433 tablet_bootstrap.cc:492] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: No bootstrap required, opened a new log
I20260927 00:53:07.672689 8433 ts_tablet_manager.cc:1397] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 00:53:07.676795 8433 raft_consensus.cc:348] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.676713 8432 raft_consensus.cc:348] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.677129 8432 raft_consensus.cc:374] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:07.677129 8433 raft_consensus.cc:374] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:07.677197 8433 raft_consensus.cc:729] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Initialized, Role: FOLLOWER
I20260927 00:53:07.677197 8432 raft_consensus.cc:729] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Initialized, Role: FOLLOWER
I20260927 00:53:07.677783 8432 consensus_queue.cc:260] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.677783 8433 consensus_queue.cc:260] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.678649 8262 heartbeater.cc:503] Master 127.0.98.62:38563 was elected leader, sending a full tablet report...
I20260927 00:53:07.679098 8432 ts_tablet_manager.cc:1428] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:53:07.679279 8402 heartbeater.cc:503] Master 127.0.98.62:38563 was elected leader, sending a full tablet report...
I20260927 00:53:07.680570 8433 ts_tablet_manager.cc:1428] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:53:07.789299 8437 raft_consensus.cc:482] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:07.789532 8437 raft_consensus.cc:504] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.790763 8437 leader_election.cc:288] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793)
I20260927 00:53:07.795511 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.791080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37601 (local address 127.0.98.3:54657)
0927 00:53:07.791341 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.791366 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.791536 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.791859 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.791863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.791890 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:07.792021 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.792028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.793040 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.793044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.793959 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.793968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.794916 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.794919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.795046 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.795182 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.795272 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.795366 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 00:53:07.795545 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.791424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54657 (local address 127.0.98.1:37601)
0927 00:53:07.791584 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.791589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.791608 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:53:07.791681 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.791684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.791736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.791822 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:07.792175 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.792908 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.794115 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.794765 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.794888 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.794973 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.795325 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.795414 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:53:07.796252 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.791372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41793 (local address 127.0.98.3:52447)
0927 00:53:07.791857 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:07.791871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:07.791969 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:07.792215 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:07.792218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:07.792234 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:07.792376 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.792384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.793260 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.793264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.794797 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:07.794806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.795823 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.795826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:07.795912 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.795930 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:53:07.796036 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:53:07.796110 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":146,"threads_started":1}
I20260927 00:53:07.796418 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.791540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52447 (local address 127.0.98.2:41793)
0927 00:53:07.791701 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.791705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.791955 (+ 250us) server_negotiation.cc:407] Connection header received
0927 00:53:07.792056 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.792059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.792109 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.792188 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:07.792529 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.793137 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.794928 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.795570 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.795670 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.795745 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.796110 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.796214 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:53:07.796340 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18c3d5210e041dcad11aada7ea0ed9a" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" is_pre_election: true
I20260927 00:53:07.796684 8076 raft_consensus.cc:2454] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 0.
I20260927 00:53:07.797206 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18c3d5210e041dcad11aada7ea0ed9a" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3" is_pre_election: true
I20260927 00:53:07.797238 8290 leader_election.cc:302] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:07.797534 8437 raft_consensus.cc:2781] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:07.797566 8216 raft_consensus.cc:2454] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 0.
I20260927 00:53:07.797627 8437 raft_consensus.cc:482] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:07.797680 8437 raft_consensus.cc:3037] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:07.798900 8437 raft_consensus.cc:504] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.799289 8437 leader_election.cc:288] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 election: Requested vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793)
I20260927 00:53:07.799747 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18c3d5210e041dcad11aada7ea0ed9a" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
I20260927 00:53:07.799782 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18c3d5210e041dcad11aada7ea0ed9a" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3"
I20260927 00:53:07.799875 8076 raft_consensus.cc:3037] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:07.799947 8216 raft_consensus.cc:3037] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:07.800877 8076 raft_consensus.cc:2454] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 1.
I20260927 00:53:07.801195 8290 leader_election.cc:302] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:07.801254 8216 raft_consensus.cc:2454] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 1.
I20260927 00:53:07.801427 8437 raft_consensus.cc:2781] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:07.801678 8437 raft_consensus.cc:686] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 1 LEADER]: Becoming Leader. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Running, Role: LEADER
I20260927 00:53:07.802030 8437 consensus_queue.cc:237] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } }
I20260927 00:53:07.805823 7919 catalog_manager.cc:5495] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 21565270df714824b8ce5bd321af42bf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "21565270df714824b8ce5bd321af42bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } health_report { overall_health: UNKNOWN } } }
W20260927 00:53:07.813936 8123 tablet.cc:2369] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:53:07.816097 8263 tablet.cc:2369] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:53:07.826846 8403 tablet.cc:2369] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:53:07.844501 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:07.841090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.0.98.3:43189)
0927 00:53:07.841253 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:07.841258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:07.841271 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:53:07.841347 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:07.841349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:07.841407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:07.841501 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:07.841719 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.842239 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.843050 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:07.843301 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:07.843389 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:07.843649 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:07.844125 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:07.844226 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:07.844302 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 00:53:07.844360 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:53:07.851315 8216 raft_consensus.cc:1261] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 21565270df714824b8ce5bd321af42bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:07.851449 8076 raft_consensus.cc:1261] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Refusing update from remote peer 21565270df714824b8ce5bd321af42bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:07.852037 8437 consensus_queue.cc:1035] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:07.852437 8440 consensus_queue.cc:1035] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:07.863451 8444 mvcc.cc:205] Tried to move back new op lower bound from 7333766708630736896 to 7333766708440027136. Current Snapshot: MvccSnapshot[applied={T|T < 7333766708630736896 or (T in {7333766708630736896})}]
I20260927 00:53:07.872624 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:07.872862 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:07.875558 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Sending DeleteTablet for 3 replicas of tablet d18c3d5210e041dcad11aada7ea0ed9a
I20260927 00:53:07.876569 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet d18c3d5210e041dcad11aada7ea0ed9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55586
I20260927 00:53:07.876760 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet d18c3d5210e041dcad11aada7ea0ed9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260927 00:53:07.876773 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet d18c3d5210e041dcad11aada7ea0ed9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32998
I20260927 00:53:07.877206 8453 tablet_replica.cc:330] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: stopping tablet replica
I20260927 00:53:07.877564 8453 raft_consensus.cc:2229] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:53:07.877907 8455 tablet_replica.cc:330] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: stopping tablet replica
I20260927 00:53:07.878021 8453 raft_consensus.cc:2258] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:07.878268 8455 raft_consensus.cc:2229] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:07.878204 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:53:07.878629 8455 raft_consensus.cc:2258] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:53:07.878684 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:07.879797 8453 ts_tablet_manager.cc:1905] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:07.880546 8455 ts_tablet_manager.cc:1905] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:07.882993 8453 ts_tablet_manager.cc:1918] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:07.883095 8453 log.cc:1196] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/d18c3d5210e041dcad11aada7ea0ed9a
I20260927 00:53:07.883402 8453 ts_tablet_manager.cc:1939] T d18c3d5210e041dcad11aada7ea0ed9a P 21565270df714824b8ce5bd321af42bf: Deleting consensus metadata
I20260927 00:53:07.883895 8455 ts_tablet_manager.cc:1918] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:07.884003 8455 log.cc:1196] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/d18c3d5210e041dcad11aada7ea0ed9a
I20260927 00:53:07.884328 8455 ts_tablet_manager.cc:1939] T d18c3d5210e041dcad11aada7ea0ed9a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting consensus metadata
I20260927 00:53:07.884652 7903 catalog_manager.cc:4841] TS 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189): tablet d18c3d5210e041dcad11aada7ea0ed9a (table test-table [id=8b71d4f445c340a0a7228de27e8bb911]) successfully deleted
I20260927 00:53:07.885288 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4873a1f7b9426ab267a60468b5185b (DEFAULT_TABLE table=test-table [id=9217a60f78764b1a94975e88226d4d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:07.885306 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4873a1f7b9426ab267a60468b5185b (DEFAULT_TABLE table=test-table [id=9217a60f78764b1a94975e88226d4d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:07.885567 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4873a1f7b9426ab267a60468b5185b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:07.885602 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4873a1f7b9426ab267a60468b5185b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:07.886982 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4873a1f7b9426ab267a60468b5185b (DEFAULT_TABLE table=test-table [id=9217a60f78764b1a94975e88226d4d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:07.887332 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4873a1f7b9426ab267a60468b5185b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:07.887898 8433 tablet_bootstrap.cc:492] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Bootstrap starting.
I20260927 00:53:07.888927 8433 tablet_bootstrap.cc:654] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:07.889768 8432 tablet_bootstrap.cc:492] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Bootstrap starting.
I20260927 00:53:07.890017 8454 tablet_replica.cc:330] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: stopping tablet replica
I20260927 00:53:07.890407 8454 raft_consensus.cc:2229] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:07.890473 8433 tablet_bootstrap.cc:492] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: No bootstrap required, opened a new log
I20260927 00:53:07.890573 8433 ts_tablet_manager.cc:1397] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:07.890767 8454 raft_consensus.cc:2258] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:07.890885 8432 tablet_bootstrap.cc:654] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:07.891109 8433 raft_consensus.cc:348] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:07.891273 8433 raft_consensus.cc:374] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:07.891304 8433 raft_consensus.cc:729] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Initialized, Role: FOLLOWER
I20260927 00:53:07.891446 8433 consensus_queue.cc:260] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:07.891726 8433 ts_tablet_manager.cc:1428] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:07.893431 8430 tablet_bootstrap.cc:492] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Bootstrap starting.
I20260927 00:53:07.894057 7906 catalog_manager.cc:4841] TS 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601): tablet d18c3d5210e041dcad11aada7ea0ed9a (table test-table [id=8b71d4f445c340a0a7228de27e8bb911]) successfully deleted
I20260927 00:53:07.894487 8430 tablet_bootstrap.cc:654] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:07.894775 8432 tablet_bootstrap.cc:492] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: No bootstrap required, opened a new log
I20260927 00:53:07.894865 8432 ts_tablet_manager.cc:1397] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 00:53:07.895418 8432 raft_consensus.cc:348] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:07.895561 8432 raft_consensus.cc:374] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:07.895605 8432 raft_consensus.cc:729] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Initialized, Role: FOLLOWER
I20260927 00:53:07.895782 8432 consensus_queue.cc:260] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:07.896461 8454 ts_tablet_manager.cc:1905] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:07.896713 8432 ts_tablet_manager.cc:1428] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:53:07.898092 8430 tablet_bootstrap.cc:492] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: No bootstrap required, opened a new log
I20260927 00:53:07.898186 8430 ts_tablet_manager.cc:1397] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 00:53:07.898857 8430 raft_consensus.cc:348] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:07.898976 8430 raft_consensus.cc:374] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:07.899005 8430 raft_consensus.cc:729] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0, State: Initialized, Role: FOLLOWER
I20260927 00:53:07.899398 8454 ts_tablet_manager.cc:1918] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:07.899497 8454 log.cc:1196] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/d18c3d5210e041dcad11aada7ea0ed9a
I20260927 00:53:07.899789 8454 ts_tablet_manager.cc:1939] T d18c3d5210e041dcad11aada7ea0ed9a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting consensus metadata
I20260927 00:53:07.900036 8430 consensus_queue.cc:260] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:07.900346 8430 ts_tablet_manager.cc:1428] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:53:07.901057 7906 catalog_manager.cc:4841] TS 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793): tablet d18c3d5210e041dcad11aada7ea0ed9a (table test-table [id=8b71d4f445c340a0a7228de27e8bb911]) successfully deleted
I20260927 00:53:08.001648 8448 raft_consensus.cc:482] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:08.001847 8448 raft_consensus.cc:504] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.002315 8448 leader_election.cc:288] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793)
I20260927 00:53:08.003088 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4873a1f7b9426ab267a60468b5185b" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" is_pre_election: true
I20260927 00:53:08.003120 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4873a1f7b9426ab267a60468b5185b" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3" is_pre_election: true
I20260927 00:53:08.003353 8076 raft_consensus.cc:2454] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 0.
I20260927 00:53:08.003353 8216 raft_consensus.cc:2454] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 0.
I20260927 00:53:08.003717 8290 leader_election.cc:302] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.003978 8448 raft_consensus.cc:2781] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:08.004051 8448 raft_consensus.cc:482] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:08.004105 8448 raft_consensus.cc:3037] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.005442 8448 raft_consensus.cc:504] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.005860 8448 leader_election.cc:288] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 election: Requested vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793)
I20260927 00:53:08.006296 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4873a1f7b9426ab267a60468b5185b" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
I20260927 00:53:08.006400 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4873a1f7b9426ab267a60468b5185b" candidate_uuid: "21565270df714824b8ce5bd321af42bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3"
I20260927 00:53:08.006454 8076 raft_consensus.cc:3037] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.006542 8216 raft_consensus.cc:3037] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.007881 8216 raft_consensus.cc:2454] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 1.
I20260927 00:53:08.007881 8076 raft_consensus.cc:2454] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21565270df714824b8ce5bd321af42bf in term 1.
I20260927 00:53:08.008316 8290 leader_election.cc:302] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 2eb226f457104f0a8cb5acb52362ebf3; no voters:
I20260927 00:53:08.008730 8448 raft_consensus.cc:2781] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:08.008863 8448 raft_consensus.cc:686] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 1 LEADER]: Becoming Leader. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Running, Role: LEADER
I20260927 00:53:08.009037 8448 consensus_queue.cc:237] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.011457 7919 catalog_manager.cc:5495] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 21565270df714824b8ce5bd321af42bf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "21565270df714824b8ce5bd321af42bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } health_report { overall_health: HEALTHY } } }
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470386, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:08.090895 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.081508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38563 (local address 127.0.0.1:57198)
0927 00:53:08.081932 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.081948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.082100 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.082348 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.082351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.082882 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:08.082985 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.082992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.083874 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.083878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.084450 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.084458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.084881 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.084884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.084970 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.085577 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:08.085589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:08.087297 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:08.089062 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:08.089075 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:08.089084 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:08.089339 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:08.089612 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:08.089615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:08.089617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:08.089701 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:08.090141 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:08.090150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:08.090283 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.090418 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.090613 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1,"thread_start_us":159,"threads_started":1}
I20260927 00:53:08.090895 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.081629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.0.98.62:38563)
0927 00:53:08.081818 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.081824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.082043 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:53:08.082190 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.082194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.082235 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.082322 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:08.083100 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.083769 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.084560 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.084779 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.084865 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.087457 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:08.087460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:08.087462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:08.087483 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:08.088909 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:08.089442 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:08.089446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:08.089448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:08.089527 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:08.089823 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:08.089826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:08.089829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:08.090039 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:08.090136 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.090474 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.090681 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":77}
I20260927 00:53:08.095820 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:53:08.096714 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:08.102907 8336 tablet_service.cc:1467] Processing CreateTablet for tablet c7feccedbf07404cb2a0015d413a54f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0190a7cdbb0f4bed90e7c5e16e7bad92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:53:08.102907 8056 tablet_service.cc:1467] Processing CreateTablet for tablet c7feccedbf07404cb2a0015d413a54f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0190a7cdbb0f4bed90e7c5e16e7bad92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:53:08.103001 8196 tablet_service.cc:1467] Processing CreateTablet for tablet c7feccedbf07404cb2a0015d413a54f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0190a7cdbb0f4bed90e7c5e16e7bad92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:53:08.103474 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7feccedbf07404cb2a0015d413a54f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.103474 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7feccedbf07404cb2a0015d413a54f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.103514 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7feccedbf07404cb2a0015d413a54f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.106379 8432 tablet_bootstrap.cc:492] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: Bootstrap starting.
I20260927 00:53:08.106875 8430 tablet_bootstrap.cc:492] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Bootstrap starting.
I20260927 00:53:08.106889 8433 tablet_bootstrap.cc:492] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: Bootstrap starting.
I20260927 00:53:08.107447 8432 tablet_bootstrap.cc:654] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.107832 8430 tablet_bootstrap.cc:654] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.107889 8433 tablet_bootstrap.cc:654] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.109139 8430 tablet_bootstrap.cc:492] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: No bootstrap required, opened a new log
I20260927 00:53:08.109257 8430 ts_tablet_manager.cc:1397] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:53:08.109498 8433 tablet_bootstrap.cc:492] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: No bootstrap required, opened a new log
I20260927 00:53:08.109581 8432 tablet_bootstrap.cc:492] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: No bootstrap required, opened a new log
I20260927 00:53:08.109599 8433 ts_tablet_manager.cc:1397] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:08.109696 8432 ts_tablet_manager.cc:1397] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:08.109763 8430 raft_consensus.cc:348] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.109910 8430 raft_consensus.cc:374] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.109949 8430 raft_consensus.cc:729] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.110054 8433 raft_consensus.cc:348] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.110090 8430 consensus_queue.cc:260] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.110190 8433 raft_consensus.cc:374] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.110234 8433 raft_consensus.cc:729] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.110191 8432 raft_consensus.cc:348] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.110306 8432 raft_consensus.cc:374] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.110363 8432 raft_consensus.cc:729] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.110404 8430 ts_tablet_manager.cc:1428] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:08.110366 8433 consensus_queue.cc:260] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.110496 8432 consensus_queue.cc:260] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.110617 8448 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.110623 8435 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.110745 8448 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.110759 8432 ts_tablet_manager.cc:1428] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:53:08.110771 8435 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.110953 8438 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.111076 8438 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.110972 8433 ts_tablet_manager.cc:1428] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:08.133002 8438 raft_consensus.cc:482] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:08.133186 8438 raft_consensus.cc:504] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.134658 8438 leader_election.cc:288] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.137688 8435 raft_consensus.cc:482] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:08.137857 8435 raft_consensus.cc:504] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.139350 8435 leader_election.cc:288] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.139605 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.135266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52643 (local address 127.0.98.1:37601)
0927 00:53:08.135432 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.135438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.135600 (+ 162us) server_negotiation.cc:407] Connection header received
0927 00:53:08.135747 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.135750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.135817 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.135899 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:08.136325 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.136971 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.138425 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.138970 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.139073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.139169 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.139375 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.139466 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:53:08.139978 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.134962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37601 (local address 127.0.98.2:52643)
0927 00:53:08.135501 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.135517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.135627 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.135941 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.135944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.135964 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:08.136217 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.136226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.137091 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.137095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.138290 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.138300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.139091 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.139094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.139234 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.139257 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.139739 (+ 482us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.139828 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":156,"threads_started":1}
I20260927 00:53:08.140057 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.135137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52783 (local address 127.0.98.3:43189)
0927 00:53:08.135433 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.135438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.135802 (+ 364us) server_negotiation.cc:407] Connection header received
0927 00:53:08.135949 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.135953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.136002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.136092 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:08.136447 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.137017 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.138631 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.139302 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.139388 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.139472 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.139827 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.139916 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:53:08.140640 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.134962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43189 (local address 127.0.98.2:52783)
0927 00:53:08.135626 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.135642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.135832 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.136107 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.136109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.136122 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:08.136312 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.136320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.137132 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.137134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.138476 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.138486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.139570 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.139573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.139682 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.139708 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.140385 (+ 677us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.140471 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":190,"thread_start_us":130,"threads_started":1}
I20260927 00:53:08.140601 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7feccedbf07404cb2a0015d413a54f0" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" is_pre_election: true
I20260927 00:53:08.140774 8076 raft_consensus.cc:2454] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 0.
I20260927 00:53:08.141149 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7feccedbf07404cb2a0015d413a54f0" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf" is_pre_election: true
I20260927 00:53:08.141192 8150 leader_election.cc:302] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb226f457104f0a8cb5acb52362ebf3, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.141431 8356 raft_consensus.cc:2454] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 0.
I20260927 00:53:08.142100 8438 raft_consensus.cc:2781] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:08.142174 8438 raft_consensus.cc:482] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:08.142211 8438 raft_consensus.cc:3037] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.143585 8438 raft_consensus.cc:504] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.144011 8438 leader_election.cc:288] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 election: Requested vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.144426 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7feccedbf07404cb2a0015d413a54f0" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
I20260927 00:53:08.144449 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7feccedbf07404cb2a0015d413a54f0" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf"
I20260927 00:53:08.144568 8076 raft_consensus.cc:3037] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.144608 8356 raft_consensus.cc:3037] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.144804 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.139908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40077 (local address 127.0.98.2:41793)
0927 00:53:08.140311 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.140315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.140630 (+ 315us) server_negotiation.cc:407] Connection header received
0927 00:53:08.141028 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.141031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.141077 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.141118 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:08.141649 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.142313 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.143591 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.144132 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.144229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.144308 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.144612 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.144688 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260927 00:53:08.145153 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.139524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41793 (local address 127.0.98.1:40077)
0927 00:53:08.140544 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.140559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.140902 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.141238 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.141241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.141259 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:08.141527 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.141534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.142441 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.142444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.143460 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.143469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.144362 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.144366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.144468 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.144494 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.144917 (+ 423us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.145005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":103,"thread_start_us":137,"threads_started":1}
I20260927 00:53:08.145826 8356 raft_consensus.cc:2454] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 1.
I20260927 00:53:08.145826 8076 raft_consensus.cc:2454] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 1.
I20260927 00:53:08.146202 8150 leader_election.cc:302] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 2eb226f457104f0a8cb5acb52362ebf3, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.146229 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7feccedbf07404cb2a0015d413a54f0" candidate_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3" is_pre_election: true
I20260927 00:53:08.146433 8216 raft_consensus.cc:2379] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 in current term 1: Already voted for candidate 2eb226f457104f0a8cb5acb52362ebf3 in this term.
I20260927 00:53:08.146540 8438 raft_consensus.cc:2781] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:08.146694 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.139703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43189 (local address 127.0.98.1:33857)
0927 00:53:08.140645 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.140660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.140819 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.141136 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.141139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.141154 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:08.141302 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.141309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.144202 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.144206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.145302 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.145311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.146251 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.146255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.146361 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.146386 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.146466 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.146537 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":3,"thread_start_us":146,"threads_started":1}
I20260927 00:53:08.146749 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.140102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33857 (local address 127.0.98.3:43189)
0927 00:53:08.140201 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.140205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.140735 (+ 530us) server_negotiation.cc:407] Connection header received
0927 00:53:08.140925 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.140928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.140974 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.141020 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:08.141421 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.141969 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.145414 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.146002 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.146095 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.146195 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.146493 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.146616 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"thread_start_us":107,"threads_started":1}
I20260927 00:53:08.146763 8438 raft_consensus.cc:686] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Running, Role: LEADER
I20260927 00:53:08.147018 8438 consensus_queue.cc:237] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.147348 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7feccedbf07404cb2a0015d413a54f0" candidate_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf" is_pre_election: true
I20260927 00:53:08.147526 8356 raft_consensus.cc:2379] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 in current term 1: Already voted for candidate 2eb226f457104f0a8cb5acb52362ebf3 in this term.
I20260927 00:53:08.147965 8007 leader_election.cc:302] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters: 21565270df714824b8ce5bd321af42bf, 2eb226f457104f0a8cb5acb52362ebf3
I20260927 00:53:08.148231 8435 raft_consensus.cc:2726] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:53:08.148236 8474 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: TxnStatusTablet state changed. Reason: New leader 2eb226f457104f0a8cb5acb52362ebf3. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.148341 8474 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:53:08.149289 8476 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:53:08.149441 8476 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:53:08.149953 7915 catalog_manager.cc:5495] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } health_report { overall_health: UNKNOWN } } }
I20260927 00:53:08.162930 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.159322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.0.98.2:41793)
0927 00:53:08.159496 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.159501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.159520 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:53:08.159728 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.159731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.159794 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.159872 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.160154 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.160695 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.161532 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.161746 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.161832 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.162102 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.162533 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.162633 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.162722 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.162776 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:53:08.162930 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.159264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41793 (local address 127.0.0.1:33014)
0927 00:53:08.159437 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.159452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.159553 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.159899 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.159902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.159913 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:53:08.160044 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.160050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.160792 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.160796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.161429 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.161437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.161840 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.161844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.161918 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.161965 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.162626 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.162639 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.162705 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.162799 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:53:08.167239 8076 raft_consensus.cc:1261] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Refusing update from remote peer 2eb226f457104f0a8cb5acb52362ebf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.167495 8356 raft_consensus.cc:1261] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Refusing update from remote peer 2eb226f457104f0a8cb5acb52362ebf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.167771 8474 consensus_queue.cc:1035] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.168056 8438 consensus_queue.cc:1035] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.169593 8435 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: TxnStatusTablet state changed. Reason: New leader 2eb226f457104f0a8cb5acb52362ebf3. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.169757 8435 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.170436 8448 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: TxnStatusTablet state changed. Reason: New leader 2eb226f457104f0a8cb5acb52362ebf3. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.170568 8448 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.172067 8474 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.172184 8474 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:53:08.174612 8443 mvcc.cc:205] Tried to move back new op lower bound from 7333766709927264256 to 7333766709853458432. Current Snapshot: MvccSnapshot[applied={T|T < 7333766709927264256}]
I20260927 00:53:08.174860 8448 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.174995 8448 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 21565270df714824b8ce5bd321af42bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.176501 8438 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.176644 8438 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 2eb226f457104f0a8cb5acb52362ebf3: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:53:08.177727 8445 mvcc.cc:205] Tried to move back new op lower bound from 7333766709927264256 to 7333766709853458432. Current Snapshot: MvccSnapshot[applied={T|T < 7333766709927264256}]
I20260927 00:53:08.177954 8435 tablet_replica.cc:439] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } } }
I20260927 00:53:08.178067 8435 tablet_replica.cc:442] T c7feccedbf07404cb2a0015d413a54f0 P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:53:08.189524 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.186308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44285 (local address 127.0.98.2:41793)
0927 00:53:08.186435 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.186439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.186650 (+ 211us) server_negotiation.cc:407] Connection header received
0927 00:53:08.186799 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.186802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.186847 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.186910 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.187183 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.187699 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.188533 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.188750 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.188832 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.189115 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.189186 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.189241 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.189328 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.189377 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:53:08.189528 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.186177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41793 (local address 127.0.98.3:44285)
0927 00:53:08.186552 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.186565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.186696 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.186929 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.186932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.186939 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:53:08.187063 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.187068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.187802 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.187805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.188430 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.188438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.188848 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.188851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.188945 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.188994 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.189256 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.189262 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.189319 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.189400 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":170,"threads_started":1}
I20260927 00:53:08.196077 8216 raft_consensus.cc:1261] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 21565270df714824b8ce5bd321af42bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.196343 8076 raft_consensus.cc:1261] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Refusing update from remote peer 21565270df714824b8ce5bd321af42bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.196704 8448 consensus_queue.cc:1035] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.196895 8437 consensus_queue.cc:1035] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.223287 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.218609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55121 (local address 127.0.98.3:43189)
0927 00:53:08.218724 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.218729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.218993 (+ 264us) server_negotiation.cc:407] Connection header received
0927 00:53:08.219226 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.219230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.219282 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.219345 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.219658 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.220548 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.221717 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.222139 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.222255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.222625 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.222736 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.222813 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.222958 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.223035 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:53:08.223767 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.218524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43189 (local address 127.0.98.2:55121)
0927 00:53:08.218903 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.218922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.219033 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.219360 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.219363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.219371 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:53:08.219496 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.219501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.220687 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.220690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.221601 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.221612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.222304 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.222307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.222391 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.222447 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.222849 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.222856 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.222932 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.223052 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":160,"threads_started":1}
I20260927 00:53:08.255084 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:08.255308 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:08.257982 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Sending DeleteTablet for 3 replicas of tablet 1d4873a1f7b9426ab267a60468b5185b
I20260927 00:53:08.258865 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4873a1f7b9426ab267a60468b5185b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55586
I20260927 00:53:08.258975 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4873a1f7b9426ab267a60468b5185b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260927 00:53:08.259181 8453 tablet_replica.cc:330] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: stopping tablet replica
I20260927 00:53:08.259322 8455 tablet_replica.cc:330] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: stopping tablet replica
I20260927 00:53:08.259315 8453 raft_consensus.cc:2229] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:53:08.259472 8455 raft_consensus.cc:2229] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:08.259472 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4873a1f7b9426ab267a60468b5185b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32998
I20260927 00:53:08.259549 8453 raft_consensus.cc:2258] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.259637 8455 raft_consensus.cc:2258] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.259685 8454 tablet_replica.cc:330] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: stopping tablet replica
I20260927 00:53:08.259796 8454 raft_consensus.cc:2229] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:08.259902 8454 raft_consensus.cc:2258] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.260706 8455 ts_tablet_manager.cc:1905] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.260993 8453 ts_tablet_manager.cc:1905] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.264169 8453 ts_tablet_manager.cc:1918] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:53:08.264243 8453 log.cc:1196] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/1d4873a1f7b9426ab267a60468b5185b
I20260927 00:53:08.264454 8453 ts_tablet_manager.cc:1939] T 1d4873a1f7b9426ab267a60468b5185b P 21565270df714824b8ce5bd321af42bf: Deleting consensus metadata
I20260927 00:53:08.265266 8454 ts_tablet_manager.cc:1905] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.265722 7903 catalog_manager.cc:4841] TS 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189): tablet 1d4873a1f7b9426ab267a60468b5185b (table test-table [id=9217a60f78764b1a94975e88226d4d1a]) successfully deleted
I20260927 00:53:08.265760 8455 ts_tablet_manager.cc:1918] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:53:08.265870 8455 log.cc:1196] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/1d4873a1f7b9426ab267a60468b5185b
I20260927 00:53:08.266152 8455 ts_tablet_manager.cc:1939] T 1d4873a1f7b9426ab267a60468b5185b P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting consensus metadata
I20260927 00:53:08.267267 7906 catalog_manager.cc:4841] TS 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601): tablet 1d4873a1f7b9426ab267a60468b5185b (table test-table [id=9217a60f78764b1a94975e88226d4d1a]) successfully deleted
I20260927 00:53:08.268410 8454 ts_tablet_manager.cc:1918] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:53:08.268514 8454 log.cc:1196] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/1d4873a1f7b9426ab267a60468b5185b
I20260927 00:53:08.268756 8454 ts_tablet_manager.cc:1939] T 1d4873a1f7b9426ab267a60468b5185b P 2eb226f457104f0a8cb5acb52362ebf3: Deleting consensus metadata
I20260927 00:53:08.269841 7906 catalog_manager.cc:4841] TS 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793): tablet 1d4873a1f7b9426ab267a60468b5185b (table test-table [id=9217a60f78764b1a94975e88226d4d1a]) successfully deleted
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:53:08.293766 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:08.285452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.0.98.62:38563)
0927 00:53:08.285598 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.285601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.285801 (+ 200us) server_negotiation.cc:407] Connection header received
0927 00:53:08.286011 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.286014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.286068 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.286151 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:08.287413 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.287999 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.289305 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.289532 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.289630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.293650 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57210: BlockingRecv error: recv got EOF from 127.0.0.1:57210 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:53:08.303047 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.295563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.98.62:38563)
0927 00:53:08.295716 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.295720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.295736 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:53:08.295795 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.295797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.295834 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.295896 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:08.296482 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.297001 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.297750 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.297952 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.298035 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.300271 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:08.300274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:08.300276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:08.300301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:08.301538 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:08.301914 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:08.301918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:08.301920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:08.301988 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:08.302223 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:08.302226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:08.302228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:08.302407 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:08.302538 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.302769 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.302884 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:53:08.307495 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:53:08.308238 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:08.316022 8336 tablet_service.cc:1467] Processing CreateTablet for tablet d20ac48b168c4610b80aa4b353494e7a (DEFAULT_TABLE table=test-table [id=c6d0cd81fe5b47dfae77c93899b664fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:08.316139 8056 tablet_service.cc:1467] Processing CreateTablet for tablet d20ac48b168c4610b80aa4b353494e7a (DEFAULT_TABLE table=test-table [id=c6d0cd81fe5b47dfae77c93899b664fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:08.316368 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20ac48b168c4610b80aa4b353494e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.316538 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20ac48b168c4610b80aa4b353494e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.317217 8196 tablet_service.cc:1467] Processing CreateTablet for tablet d20ac48b168c4610b80aa4b353494e7a (DEFAULT_TABLE table=test-table [id=c6d0cd81fe5b47dfae77c93899b664fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:08.317536 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20ac48b168c4610b80aa4b353494e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.319186 8433 tablet_bootstrap.cc:492] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Bootstrap starting.
I20260927 00:53:08.320065 8433 tablet_bootstrap.cc:654] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.322885 8430 tablet_bootstrap.cc:492] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Bootstrap starting.
I20260927 00:53:08.324018 8430 tablet_bootstrap.cc:654] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.324684 8432 tablet_bootstrap.cc:492] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Bootstrap starting.
I20260927 00:53:08.325848 8432 tablet_bootstrap.cc:654] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.325973 8430 tablet_bootstrap.cc:492] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: No bootstrap required, opened a new log
I20260927 00:53:08.326052 8430 ts_tablet_manager.cc:1397] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:08.326545 8430 raft_consensus.cc:348] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.326664 8430 raft_consensus.cc:374] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.326689 8430 raft_consensus.cc:729] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.326795 8430 consensus_queue.cc:260] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.327026 8430 ts_tablet_manager.cc:1428] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:53:08.328683 8432 tablet_bootstrap.cc:492] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: No bootstrap required, opened a new log
I20260927 00:53:08.328800 8432 ts_tablet_manager.cc:1397] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:53:08.329437 8433 tablet_bootstrap.cc:492] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: No bootstrap required, opened a new log
I20260927 00:53:08.329414 8432 raft_consensus.cc:348] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.329530 8433 ts_tablet_manager.cc:1397] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260927 00:53:08.329564 8432 raft_consensus.cc:374] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.329609 8432 raft_consensus.cc:729] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.329756 8432 consensus_queue.cc:260] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.329912 8433 raft_consensus.cc:348] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.330024 8433 raft_consensus.cc:374] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.330035 8432 ts_tablet_manager.cc:1428] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:53:08.330081 8433 raft_consensus.cc:729] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.330212 8433 consensus_queue.cc:260] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.330480 8433 ts_tablet_manager.cc:1428] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:08.386974 8489 raft_consensus.cc:482] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:08.387231 8489 raft_consensus.cc:504] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.387784 8489 leader_election.cc:288] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.388355 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20ac48b168c4610b80aa4b353494e7a" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" is_pre_election: true
I20260927 00:53:08.388438 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20ac48b168c4610b80aa4b353494e7a" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf" is_pre_election: true
I20260927 00:53:08.388531 8076 raft_consensus.cc:2454] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 0.
I20260927 00:53:08.388620 8355 raft_consensus.cc:2454] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 0.
I20260927 00:53:08.388940 8150 leader_election.cc:302] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb226f457104f0a8cb5acb52362ebf3, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.389168 8489 raft_consensus.cc:2781] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:08.389242 8489 raft_consensus.cc:482] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:08.389287 8489 raft_consensus.cc:3037] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.390369 8489 raft_consensus.cc:504] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.390754 8489 leader_election.cc:288] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 election: Requested vote from peers 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.391219 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20ac48b168c4610b80aa4b353494e7a" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf"
I20260927 00:53:08.391294 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20ac48b168c4610b80aa4b353494e7a" candidate_uuid: "2eb226f457104f0a8cb5acb52362ebf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0"
I20260927 00:53:08.391373 8355 raft_consensus.cc:3037] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.391398 8076 raft_consensus.cc:3037] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.392395 8076 raft_consensus.cc:2454] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 1.
I20260927 00:53:08.392758 8355 raft_consensus.cc:2454] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb226f457104f0a8cb5acb52362ebf3 in term 1.
I20260927 00:53:08.392769 8150 leader_election.cc:302] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb226f457104f0a8cb5acb52362ebf3, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.392952 8489 raft_consensus.cc:2781] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:08.393040 8489 raft_consensus.cc:686] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Running, Role: LEADER
I20260927 00:53:08.393191 8489 consensus_queue.cc:237] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.395421 7914 catalog_manager.cc:5495] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2eb226f457104f0a8cb5acb52362ebf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } health_report { overall_health: UNKNOWN } } }
I20260927 00:53:08.421243 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.417525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.0.98.2:41793)
0927 00:53:08.417699 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.417703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.417721 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:53:08.417804 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.417807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.417853 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.417929 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.418230 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.418751 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.419866 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.420143 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.420250 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.420560 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.420634 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.420690 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.420987 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.421072 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:53:08.424383 8076 raft_consensus.cc:1261] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Refusing update from remote peer 2eb226f457104f0a8cb5acb52362ebf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.424396 8355 raft_consensus.cc:1261] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Refusing update from remote peer 2eb226f457104f0a8cb5acb52362ebf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.424932 8474 consensus_queue.cc:1035] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.425213 8489 consensus_queue.cc:1035] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.437228 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:08.437419 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:08.440068 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Sending DeleteTablet for 3 replicas of tablet d20ac48b168c4610b80aa4b353494e7a
I20260927 00:53:08.440817 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet d20ac48b168c4610b80aa4b353494e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55586
I20260927 00:53:08.440838 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet d20ac48b168c4610b80aa4b353494e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260927 00:53:08.441025 8453 tablet_replica.cc:330] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: stopping tablet replica
I20260927 00:53:08.441061 8455 tablet_replica.cc:330] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: stopping tablet replica
I20260927 00:53:08.441166 8453 raft_consensus.cc:2229] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:08.441179 8455 raft_consensus.cc:2229] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:08.441327 8453 raft_consensus.cc:2258] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.441340 8455 raft_consensus.cc:2258] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.441699 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet d20ac48b168c4610b80aa4b353494e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32998
I20260927 00:53:08.441910 8454 tablet_replica.cc:330] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: stopping tablet replica
I20260927 00:53:08.442021 8454 raft_consensus.cc:2229] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:53:08.442122 8453 ts_tablet_manager.cc:1905] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.442400 8454 raft_consensus.cc:2258] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.442327 7915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:53:08.442799 7915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:08.443444 8454 ts_tablet_manager.cc:1905] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.445171 8453 ts_tablet_manager.cc:1918] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:08.445259 8453 log.cc:1196] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/d20ac48b168c4610b80aa4b353494e7a
I20260927 00:53:08.445523 8453 ts_tablet_manager.cc:1939] T d20ac48b168c4610b80aa4b353494e7a P 21565270df714824b8ce5bd321af42bf: Deleting consensus metadata
I20260927 00:53:08.446600 7903 catalog_manager.cc:4841] TS 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189): tablet d20ac48b168c4610b80aa4b353494e7a (table test-table [id=c6d0cd81fe5b47dfae77c93899b664fa]) successfully deleted
I20260927 00:53:08.447176 8454 ts_tablet_manager.cc:1918] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:08.447271 8454 log.cc:1196] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/d20ac48b168c4610b80aa4b353494e7a
I20260927 00:53:08.447535 8454 ts_tablet_manager.cc:1939] T d20ac48b168c4610b80aa4b353494e7a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting consensus metadata
I20260927 00:53:08.448392 7906 catalog_manager.cc:4841] TS 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793): tablet d20ac48b168c4610b80aa4b353494e7a (table test-table [id=c6d0cd81fe5b47dfae77c93899b664fa]) successfully deleted
I20260927 00:53:08.449321 8455 ts_tablet_manager.cc:1905] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.450456 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 0c635732499741e2b2d08476ea32791a (DEFAULT_TABLE table=test-table [id=9e253f4661da4167a39bc43e3c2b9b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:08.450773 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c635732499741e2b2d08476ea32791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.451953 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 0c635732499741e2b2d08476ea32791a (DEFAULT_TABLE table=test-table [id=9e253f4661da4167a39bc43e3c2b9b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:08.452309 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c635732499741e2b2d08476ea32791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.452489 8455 ts_tablet_manager.cc:1918] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:08.452561 8455 log.cc:1196] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/d20ac48b168c4610b80aa4b353494e7a
I20260927 00:53:08.452847 8455 ts_tablet_manager.cc:1939] T d20ac48b168c4610b80aa4b353494e7a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting consensus metadata
I20260927 00:53:08.453608 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 0c635732499741e2b2d08476ea32791a (DEFAULT_TABLE table=test-table [id=9e253f4661da4167a39bc43e3c2b9b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:08.453917 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c635732499741e2b2d08476ea32791a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:08.454936 8433 tablet_bootstrap.cc:492] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Bootstrap starting.
I20260927 00:53:08.455782 8433 tablet_bootstrap.cc:654] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.456319 8432 tablet_bootstrap.cc:492] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Bootstrap starting.
I20260927 00:53:08.456640 7906 catalog_manager.cc:4841] TS 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601): tablet d20ac48b168c4610b80aa4b353494e7a (table test-table [id=c6d0cd81fe5b47dfae77c93899b664fa]) successfully deleted
I20260927 00:53:08.456887 8430 tablet_bootstrap.cc:492] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Bootstrap starting.
I20260927 00:53:08.457229 8432 tablet_bootstrap.cc:654] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.457923 8430 tablet_bootstrap.cc:654] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:08.458426 8433 tablet_bootstrap.cc:492] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: No bootstrap required, opened a new log
I20260927 00:53:08.458540 8433 ts_tablet_manager.cc:1397] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:53:08.458614 8432 tablet_bootstrap.cc:492] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: No bootstrap required, opened a new log
I20260927 00:53:08.458709 8432 ts_tablet_manager.cc:1397] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:53:08.458987 8433 raft_consensus.cc:348] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.459121 8433 raft_consensus.cc:374] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.459195 8433 raft_consensus.cc:729] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21565270df714824b8ce5bd321af42bf, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.459105 8432 raft_consensus.cc:348] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.459241 8432 raft_consensus.cc:374] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.459278 8432 raft_consensus.cc:729] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb226f457104f0a8cb5acb52362ebf3, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.459319 8433 consensus_queue.cc:260] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.459369 8430 tablet_bootstrap.cc:492] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: No bootstrap required, opened a new log
I20260927 00:53:08.459393 8432 consensus_queue.cc:260] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.459476 8430 ts_tablet_manager.cc:1397] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:53:08.459569 8433 ts_tablet_manager.cc:1428] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:53:08.459669 8432 ts_tablet_manager.cc:1428] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:53:08.460012 8430 raft_consensus.cc:348] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.460147 8430 raft_consensus.cc:374] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:08.460187 8430 raft_consensus.cc:729] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0, State: Initialized, Role: FOLLOWER
I20260927 00:53:08.460337 8430 consensus_queue.cc:260] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.460616 8430 ts_tablet_manager.cc:1428] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:53:08.656347 8435 raft_consensus.cc:482] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:08.656524 8435 raft_consensus.cc:504] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.656956 8435 leader_election.cc:288] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.657605 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c635732499741e2b2d08476ea32791a" candidate_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf" is_pre_election: true
I20260927 00:53:08.657714 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c635732499741e2b2d08476ea32791a" candidate_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3" is_pre_election: true
I20260927 00:53:08.657778 8355 raft_consensus.cc:2454] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 in term 0.
I20260927 00:53:08.657866 8216 raft_consensus.cc:2454] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 in term 0.
I20260927 00:53:08.658316 8007 leader_election.cc:302] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 2eb226f457104f0a8cb5acb52362ebf3, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.658631 8435 raft_consensus.cc:2781] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:08.658702 8435 raft_consensus.cc:482] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:08.658730 8435 raft_consensus.cc:3037] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.659929 8435 raft_consensus.cc:504] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.660359 8435 leader_election.cc:288] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [CANDIDATE]: Term 1 election: Requested vote from peers 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793), 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189)
I20260927 00:53:08.660833 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c635732499741e2b2d08476ea32791a" candidate_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb226f457104f0a8cb5acb52362ebf3"
I20260927 00:53:08.660907 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c635732499741e2b2d08476ea32791a" candidate_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21565270df714824b8ce5bd321af42bf"
I20260927 00:53:08.661062 8355 raft_consensus.cc:3037] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.661051 8216 raft_consensus.cc:3037] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:08.662544 8355 raft_consensus.cc:2454] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 in term 1.
I20260927 00:53:08.662544 8216 raft_consensus.cc:2454] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 in term 1.
I20260927 00:53:08.662977 8007 leader_election.cc:302] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21565270df714824b8ce5bd321af42bf, 2eb226f457104f0a8cb5acb52362ebf3, 5a9a87bfeb0d4a48ac0eb938f8cb8ee0; no voters:
I20260927 00:53:08.663405 8435 raft_consensus.cc:2781] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:08.663614 8435 raft_consensus.cc:686] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 LEADER]: Becoming Leader. State: Replica: 5a9a87bfeb0d4a48ac0eb938f8cb8ee0, State: Running, Role: LEADER
I20260927 00:53:08.663925 8435 consensus_queue.cc:237] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } }
I20260927 00:53:08.667011 7918 catalog_manager.cc:5495] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a87bfeb0d4a48ac0eb938f8cb8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 } health_report { overall_health: UNKNOWN } } }
I20260927 00:53:08.707906 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.703972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.0.98.1:37601)
0927 00:53:08.704222 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.704226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.704238 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:53:08.704335 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.704338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.704402 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.704524 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.704717 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.705275 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.706127 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.706411 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.706493 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.706738 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.707480 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.707578 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.707673 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.707729 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260927 00:53:08.719714 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.716007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57363 (local address 127.0.98.2:41793)
0927 00:53:08.716172 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.716176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.716250 (+ 74us) server_negotiation.cc:407] Connection header received
0927 00:53:08.716430 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.716432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.716492 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.716585 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.716871 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.717525 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.718509 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.718736 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.718829 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.719288 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.719379 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.719436 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.719521 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.719572 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:53:08.719734 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.715826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41793 (local address 127.0.98.1:57363)
0927 00:53:08.716158 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.716171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.716278 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.716605 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.716607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.716612 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:53:08.716728 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.716732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.717636 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.717638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.718370 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.718382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.718875 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.718878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.718958 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.719016 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.719452 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.719457 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.719505 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.719577 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260927 00:53:08.727581 8216 raft_consensus.cc:1261] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.728058 8355 raft_consensus.cc:1261] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Refusing update from remote peer 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:08.728189 8512 consensus_queue.cc:1035] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb226f457104f0a8cb5acb52362ebf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.728514 8435 consensus_queue.cc:1035] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21565270df714824b8ce5bd321af42bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:08.752686 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.749187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37601 (local address 127.0.98.2:51215)
0927 00:53:08.749524 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:08.749541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:08.749655 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:08.749923 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:08.749927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:08.749937 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:53:08.750047 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.750054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.750810 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.750814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.751487 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:08.751497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.751900 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.751903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:08.751983 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.752013 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.752388 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.752393 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:53:08.752446 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 00:53:08.752535 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":85,"threads_started":1}
I20260927 00:53:08.752668 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:08.749308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51215 (local address 127.0.98.1:37601)
0927 00:53:08.749451 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:08.749456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:08.749678 (+ 222us) server_negotiation.cc:407] Connection header received
0927 00:53:08.749741 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:08.749758 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:08.749815 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:08.749898 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:08.750198 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.750708 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.751594 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:08.751805 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:08.751890 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:08.752118 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:08.752188 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:08.752252 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:08.752471 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:53:08.752525 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:53:08.780205 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:08.780524 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:08.783273 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62bb933a66c044c582ff3b61c6678a00: Sending DeleteTablet for 3 replicas of tablet 0c635732499741e2b2d08476ea32791a
I20260927 00:53:08.783957 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c635732499741e2b2d08476ea32791a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39598
I20260927 00:53:08.784071 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c635732499741e2b2d08476ea32791a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55586
I20260927 00:53:08.784147 8455 tablet_replica.cc:330] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: stopping tablet replica
I20260927 00:53:08.784225 8453 tablet_replica.cc:330] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: stopping tablet replica
I20260927 00:53:08.784252 8455 raft_consensus.cc:2229] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:53:08.784317 8453 raft_consensus.cc:2229] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:08.784428 8453 raft_consensus.cc:2258] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.784570 8455 raft_consensus.cc:2258] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.784931 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c635732499741e2b2d08476ea32791a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32998
I20260927 00:53:08.785213 8454 tablet_replica.cc:330] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: stopping tablet replica
I20260927 00:53:08.785261 8453 ts_tablet_manager.cc:1905] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.785454 8454 raft_consensus.cc:2229] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:08.785619 8454 raft_consensus.cc:2258] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:08.786648 8455 ts_tablet_manager.cc:1905] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.786937 8454 ts_tablet_manager.cc:1905] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:08.787807 8453 ts_tablet_manager.cc:1918] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:53:08.787886 8453 log.cc:1196] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/0c635732499741e2b2d08476ea32791a
I20260927 00:53:08.788120 8453 ts_tablet_manager.cc:1939] T 0c635732499741e2b2d08476ea32791a P 21565270df714824b8ce5bd321af42bf: Deleting consensus metadata
I20260927 00:53:08.789122 8455 ts_tablet_manager.cc:1918] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:53:08.789177 8455 log.cc:1196] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/0c635732499741e2b2d08476ea32791a
I20260927 00:53:08.789333 8455 ts_tablet_manager.cc:1939] T 0c635732499741e2b2d08476ea32791a P 5a9a87bfeb0d4a48ac0eb938f8cb8ee0: Deleting consensus metadata
I20260927 00:53:08.790087 7906 catalog_manager.cc:4841] TS 5a9a87bfeb0d4a48ac0eb938f8cb8ee0 (127.0.98.1:37601): tablet 0c635732499741e2b2d08476ea32791a (table test-table [id=9e253f4661da4167a39bc43e3c2b9b69]) successfully deleted
I20260927 00:53:08.790160 8454 ts_tablet_manager.cc:1918] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:53:08.790287 8454 log.cc:1196] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/0c635732499741e2b2d08476ea32791a
I20260927 00:53:08.790577 8454 ts_tablet_manager.cc:1939] T 0c635732499741e2b2d08476ea32791a P 2eb226f457104f0a8cb5acb52362ebf3: Deleting consensus metadata
I20260927 00:53:08.791314 7903 catalog_manager.cc:4841] TS 21565270df714824b8ce5bd321af42bf (127.0.98.3:43189): tablet 0c635732499741e2b2d08476ea32791a (table test-table [id=9e253f4661da4167a39bc43e3c2b9b69]) successfully deleted
I20260927 00:53:08.791564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 7991
I20260927 00:53:08.793344 7906 catalog_manager.cc:4841] TS 2eb226f457104f0a8cb5acb52362ebf3 (127.0.98.2:41793): tablet 0c635732499741e2b2d08476ea32791a (table test-table [id=9e253f4661da4167a39bc43e3c2b9b69]) successfully deleted
I20260927 00:53:08.799592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 8131
I20260927 00:53:08.806939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 8271
I20260927 00:53:08.813202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 7887
2026-09-27T00:53:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3366 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[8528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[8528](info): set up 2 sockets
Sep 27 00:53:08 dist-test-slave-9fh2 krb5kdc[8528](info): commencing operation
krb5kdc: starting...
W20260927 00:53:09.804062 392 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:53:09 dist-test-slave-9fh2 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:53:09.939703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:10.073228 8541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:10.073629 8541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:10.073676 8541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:10.078156 8541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:10.078246 8541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:10.078266 8541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:10.078286 8541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:10.078302 8541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:10.084606 8541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:10.086061 8541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:10.087361 8541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:10.097196 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:10.097218 8547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:10.097303 8541 server_base.cc:1034] running on GCE node
W20260927 00:53:10.097204 8549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:10.098184 8541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:10.099527 8541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:10.100701 8541 hybrid_clock.cc:648] HybridClock initialized: now 1790470390100679 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T00:53:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1294 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:53:10 dist-test-slave-9fh2 krb5kdc[8554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:53:10 dist-test-slave-9fh2 krb5kdc[8554](info): set up 2 sockets
Sep 27 00:53:10 dist-test-slave-9fh2 krb5kdc[8554](info): commencing operation
krb5kdc: starting...
W20260927 00:53:11.076161 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:53:11.214527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:11.345018 8567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:11.345494 8567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:11.345566 8567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:11.350193 8567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:11.350306 8567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:11.350335 8567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:11.350358 8567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:11.350378 8567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:11.356763 8567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:11.358323 8567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:11.359746 8567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:11.370543 8573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:11.370550 8575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:11.370543 8572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:11.371196 8567 server_base.cc:1034] running on GCE node
I20260927 00:53:11.372009 8567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:11.373479 8567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:11.374668 8567 hybrid_clock.cc:648] HybridClock initialized: now 1790470391374650 us; error 39 us; skew 500 ppm
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:11.377960 8567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:53:11.379098 8567 webserver.cc:460] Webserver started at http://127.0.98.62:41489/ using document root <none> and password file <none>
I20260927 00:53:11.379810 8567 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:11.379876 8567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:11.380091 8567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:11.382581 8567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c2bc12ff039a4892bbc9c0e91068566d"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "65cb4c16b4c2da18f4bcf3e8287ab8b6"
server_key_iv: "a5591f7f8a761fb6e6a98f4837bf0212"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.383244 8567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2bc12ff039a4892bbc9c0e91068566d"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "65cb4c16b4c2da18f4bcf3e8287ab8b6"
server_key_iv: "a5591f7f8a761fb6e6a98f4837bf0212"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.388252 8567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:53:11.391448 8582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.393183 8567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:11.393396 8567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c2bc12ff039a4892bbc9c0e91068566d"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "65cb4c16b4c2da18f4bcf3e8287ab8b6"
server_key_iv: "a5591f7f8a761fb6e6a98f4837bf0212"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.393527 8567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:11.418905 8567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:11.423261 8567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:11.423468 8567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:11.431825 8567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46517
I20260927 00:53:11.431974 8634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46517 every 8 connection(s)
I20260927 00:53:11.433038 8567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:11.436228 8635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:11.441907 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Bootstrap starting.
I20260927 00:53:11.442858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 8567
I20260927 00:53:11.442991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:53:11.443354 392 external_mini_cluster.cc:1428] Setting key 4fe1663c9ee8f032de96d9c20250929c
I20260927 00:53:11.445276 8635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:11.445998 8635 log.cc:824] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:11.448207 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: No bootstrap required, opened a new log
I20260927 00:53:11.452708 8635 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:11.453001 8635 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:11.453060 8635 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bc12ff039a4892bbc9c0e91068566d, State: Initialized, Role: FOLLOWER
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:53:11.453519 8635 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:11.453672 8635 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:11.453734 8635 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:11.453827 8635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:11.454871 8635 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:11.455371 8635 leader_election.cc:302] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2bc12ff039a4892bbc9c0e91068566d; no voters:
I20260927 00:53:11.455736 8635 leader_election.cc:288] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:53:11.455853 8640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:11.456080 8640 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 LEADER]: Becoming Leader. State: Replica: c2bc12ff039a4892bbc9c0e91068566d, State: Running, Role: LEADER
I20260927 00:53:11.456538 8640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:11.457247 8635 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:11.457818 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2bc12ff039a4892bbc9c0e91068566d. Latest consensus state: current_term: 1 leader_uuid: "c2bc12ff039a4892bbc9c0e91068566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } }
I20260927 00:53:11.458666 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:11.458551 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2bc12ff039a4892bbc9c0e91068566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } }
I20260927 00:53:11.458989 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:11.459349 8650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:11.460737 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:11.444695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.0.98.62:46517)
0927 00:53:11.445154 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:11.445175 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:11.445220 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:53:11.446111 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:11.446140 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:11.446600 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:11.446984 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:11.447913 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.449084 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.450306 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.450895 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.451054 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:11.454296 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:11.454304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:11.454329 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:11.454368 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:11.456488 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:11.457024 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:11.457031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:11.457038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:11.457169 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:11.457440 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:11.457444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:11.457446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:11.458057 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:11.459054 (+ 997us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:11.459259 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 00:53:11.459610 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
I20260927 00:53:11.463094 8650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:11.468495 8650 catalog_manager.cc:1347] Generated new cluster ID: cff10f85a4ef4c2f98bc13c9ddffec37
I20260927 00:53:11.468583 8650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:11.478761 8650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:53:11.479770 8650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:11.489727 8650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Generated new TSK 0
I20260927 00:53:11.490418 8650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:53:11.525985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46517
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:11.648274 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:11.648551 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:11.648595 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:11.652779 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:11.652851 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:11.652912 8661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:11.658507 8661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:11.659719 8661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:11.660770 8661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:11.671576 8669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:11.671566 8666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:11.671641 8661 server_base.cc:1034] running on GCE node
W20260927 00:53:11.671566 8667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:11.672442 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:11.673295 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:11.674476 8661 hybrid_clock.cc:648] HybridClock initialized: now 1790470391674451 us; error 45 us; skew 500 ppm
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:11.677824 8661 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:53:11.678993 8661 webserver.cc:460] Webserver started at http://127.0.98.1:41629/ using document root <none> and password file <none>
I20260927 00:53:11.679713 8661 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:11.679785 8661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:11.679993 8661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:11.682575 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "c23f0a75b88e4310a28e3d2590a2ecac"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "63a2612e85f8fce473bf67561626420d"
server_key_iv: "93cd87323b97119bfa627a6a95028efe"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.683341 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c23f0a75b88e4310a28e3d2590a2ecac"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "63a2612e85f8fce473bf67561626420d"
server_key_iv: "93cd87323b97119bfa627a6a95028efe"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.688359 8661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:53:11.691896 8676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.693663 8661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:53:11.693842 8661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "c23f0a75b88e4310a28e3d2590a2ecac"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "63a2612e85f8fce473bf67561626420d"
server_key_iv: "93cd87323b97119bfa627a6a95028efe"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.694006 8661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:11.712668 8661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:11.717037 8661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:11.717233 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:11.717937 8661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:11.718971 8661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:11.719033 8661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.719077 8661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:11.719103 8661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.730106 8661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45783
I20260927 00:53:11.730149 8789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45783 every 8 connection(s)
I20260927 00:53:11.731331 8661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:11.734019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 8661
I20260927 00:53:11.734140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:53:11.734529 392 external_mini_cluster.cc:1428] Setting key 49884b04afd2d6ce59954d7c3c0c6827
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:53:11.749722 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:11.733500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34279 (local address 127.0.98.62:46517)
0927 00:53:11.733745 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:11.733752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:11.734848 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 00:53:11.736669 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:11.736671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:11.736718 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:11.736801 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:11.738693 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.739327 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.740609 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.740862 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.740966 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:11.744584 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:11.744588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:11.744591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:11.744643 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:11.746720 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:11.747474 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:11.747478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:11.747496 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:11.747592 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:11.748068 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:11.748074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:11.748076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:11.748416 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:11.748548 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:11.749394 (+ 846us) server_negotiation.cc:299] Negotiation successful
0927 00:53:11.749547 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 00:53:11.750797 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:11.733849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:34279)
0927 00:53:11.734595 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:11.734671 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:11.736405 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:11.737060 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:11.737071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:11.737648 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:11.738508 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:11.738533 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.739454 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.739458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:11.740478 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:11.740489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.741031 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.741035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:11.741226 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:11.742011 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:11.742035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:11.744339 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:11.746876 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:11.746892 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:11.746916 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:11.747338 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:11.747693 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:11.747696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:11.747698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:11.747858 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:11.748670 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:11.748681 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:11.749107 (+ 426us) client_negotiation.cc:769] Sending connection context
0927 00:53:11.749304 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:53:11.749703 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":160,"threads_started":1}
I20260927 00:53:11.752326 8790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46517
I20260927 00:53:11.752725 8790 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:11.753513 8790 heartbeater.cc:511] Master 127.0.98.62:46517 requested a full tablet report, sending...
I20260927 00:53:11.755576 8599 ts_manager.cc:194] Registered new tserver with Master: c23f0a75b88e4310a28e3d2590a2ecac (127.0.98.1:45783)
I20260927 00:53:11.756752 8599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:53:11.767504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46517
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:11.896606 8795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:11.896962 8795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:11.897032 8795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:11.901705 8795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:11.901795 8795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:11.901873 8795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:11.908175 8795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:11.909615 8795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:11.910872 8795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:11.922459 8803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:11.922709 8795 server_base.cc:1034] running on GCE node
W20260927 00:53:11.922454 8801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:11.922454 8800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:11.923489 8795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:11.924328 8795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:11.925498 8795 hybrid_clock.cc:648] HybridClock initialized: now 1790470391925467 us; error 48 us; skew 500 ppm
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:11.928676 8795 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:53:11.929723 8795 webserver.cc:460] Webserver started at http://127.0.98.2:43463/ using document root <none> and password file <none>
I20260927 00:53:11.930335 8795 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:11.930405 8795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:11.930601 8795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:11.932854 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "9cd8cee56a6c42a293f48db66b954971"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "f701dc5ca0b0499a04b94ea3227dc4b4"
server_key_iv: "6fbbfbedd6052c48467e846af691bb48"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.933434 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9cd8cee56a6c42a293f48db66b954971"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "f701dc5ca0b0499a04b94ea3227dc4b4"
server_key_iv: "6fbbfbedd6052c48467e846af691bb48"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.937762 8795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:53:11.940712 8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.942293 8795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:53:11.942420 8795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "9cd8cee56a6c42a293f48db66b954971"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "f701dc5ca0b0499a04b94ea3227dc4b4"
server_key_iv: "6fbbfbedd6052c48467e846af691bb48"
server_key_version: "encryptionkey@0"
I20260927 00:53:11.942525 8795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:11.954645 8795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:11.958154 8795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:11.958333 8795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:11.959040 8795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:11.960085 8795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:11.960150 8795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.960203 8795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:11.960247 8795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:11.969866 8795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33267
I20260927 00:53:11.969892 8923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33267 every 8 connection(s)
I20260927 00:53:11.970840 8795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:11.976722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 8795
I20260927 00:53:11.976887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:53:11.977231 392 external_mini_cluster.cc:1428] Setting key dd2bf6768a9a63b02e9364890857ee9e
Sep 27 00:53:11 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:53:11.990195 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:11.973224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47723 (local address 127.0.98.62:46517)
0927 00:53:11.973441 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:11.973446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:11.974458 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 00:53:11.976170 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:11.976174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:11.976235 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:11.976337 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:11.978774 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.979461 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.981386 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.981659 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.981795 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:11.985388 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:11.985392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:11.985394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:11.985423 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:11.987058 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:11.988000 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:11.988004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:11.988005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:11.988098 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:11.988647 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:11.988650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:11.988651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:11.988928 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:11.989043 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:11.989876 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 00:53:11.990009 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 00:53:11.991436 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:11.973659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:47723)
0927 00:53:11.974246 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:11.974306 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:11.975895 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:11.976632 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:11.976644 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:11.977310 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:11.978114 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:11.978136 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.979622 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.979626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:11.981204 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:11.981221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:11.981908 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:11.981915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:11.982157 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:11.982940 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:11.982964 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:11.985133 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:11.987284 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:11.987293 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:11.987322 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:11.987851 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:11.988250 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:11.988253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:11.988255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:11.988497 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:11.989067 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:11.989076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:11.989562 (+ 486us) client_negotiation.cc:769] Sending connection context
0927 00:53:11.989818 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 00:53:11.990205 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:53:11.993139 8924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46517
I20260927 00:53:11.993486 8924 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:11.994227 8924 heartbeater.cc:511] Master 127.0.98.62:46517 requested a full tablet report, sending...
I20260927 00:53:11.995771 8599 ts_manager.cc:194] Registered new tserver with Master: 9cd8cee56a6c42a293f48db66b954971 (127.0.98.2:33267)
I20260927 00:53:11.996402 8599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:53:12.011762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46517
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:12.140957 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:12.141311 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:12.141381 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:12.145764 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:12.145849 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:12.145936 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:12.151968 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:12.153420 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:12.154697 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:12.166157 8934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.166133 8935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.166133 8937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:12.166534 8929 server_base.cc:1034] running on GCE node
I20260927 00:53:12.167172 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:12.168020 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:12.169236 8929 hybrid_clock.cc:648] HybridClock initialized: now 1790470392169213 us; error 40 us; skew 500 ppm
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:12.172672 8929 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:53:12.173727 8929 webserver.cc:460] Webserver started at http://127.0.98.3:34763/ using document root <none> and password file <none>
I20260927 00:53:12.174314 8929 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:12.174384 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:12.174588 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:12.176780 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "f73e33edfa174c4486e58e0344d075f7"
format_stamp: "Formatted at 2026-09-27 00:53:12 on dist-test-slave-9fh2"
server_key: "1eaf096ccb6ca259cd951b4d18c29d97"
server_key_iv: "59297f13571ff74dd75e607fcb1a043a"
server_key_version: "encryptionkey@0"
I20260927 00:53:12.177378 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f73e33edfa174c4486e58e0344d075f7"
format_stamp: "Formatted at 2026-09-27 00:53:12 on dist-test-slave-9fh2"
server_key: "1eaf096ccb6ca259cd951b4d18c29d97"
server_key_iv: "59297f13571ff74dd75e607fcb1a043a"
server_key_version: "encryptionkey@0"
I20260927 00:53:12.181816 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:53:12.184814 8944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.186383 8929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:53:12.186548 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "f73e33edfa174c4486e58e0344d075f7"
format_stamp: "Formatted at 2026-09-27 00:53:12 on dist-test-slave-9fh2"
server_key: "1eaf096ccb6ca259cd951b4d18c29d97"
server_key_iv: "59297f13571ff74dd75e607fcb1a043a"
server_key_version: "encryptionkey@0"
I20260927 00:53:12.186668 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:12.204821 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:12.208001 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:12.208184 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:12.208773 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:12.209628 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:12.209679 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.209723 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:12.209757 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.220643 8929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34859
I20260927 00:53:12.220726 9057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34859 every 8 connection(s)
I20260927 00:53:12.221843 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:12.230528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 8929
I20260927 00:53:12.230664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:53:12.230993 392 external_mini_cluster.cc:1428] Setting key 34852346e1468873e7bf316732e8b7bd
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:53:12.238637 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:12.224216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57367 (local address 127.0.98.62:46517)
0927 00:53:12.224351 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:12.224355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:12.225807 (+ 1452us) server_negotiation.cc:407] Connection header received
0927 00:53:12.227678 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:12.227682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:12.227746 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:12.227823 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:12.229130 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.229664 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.230937 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.231217 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.231302 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.234518 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:12.234522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:12.234524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:12.234552 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:12.236094 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:12.236825 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:12.236828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:12.236829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:12.236924 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:12.237397 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:12.237399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:12.237401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:12.237598 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:12.237691 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:12.238383 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 00:53:12.238499 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:53:12.239874 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:12.224908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:57367)
0927 00:53:12.225592 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:12.225655 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:12.227424 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:12.228031 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:12.228038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:12.228440 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:12.228973 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.228986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.229784 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.229786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.230824 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.230835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.231380 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.231384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.231584 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.232416 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:12.232461 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:12.234307 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:12.236269 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:12.236277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:12.236303 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:12.236681 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:12.237084 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:12.237087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:12.237089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:12.237279 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:12.237710 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:12.237717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:12.238101 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 00:53:12.238310 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:53:12.238679 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":181,"threads_started":1}
I20260927 00:53:12.241377 9058 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46517
I20260927 00:53:12.241717 9058 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:12.242384 9058 heartbeater.cc:511] Master 127.0.98.62:46517 requested a full tablet report, sending...
I20260927 00:53:12.243688 8599 ts_manager.cc:194] Registered new tserver with Master: f73e33edfa174c4486e58e0344d075f7 (127.0.98.3:34859)
I20260927 00:53:12.244356 8599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57367
I20260927 00:53:12.245772 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:53:12.330524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 8661
I20260927 00:53:12.335918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 8795
I20260927 00:53:12.344303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 8929
I20260927 00:53:12.350472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 8567
I20260927 00:53:12.357612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--webserver_interface=127.0.98.62
--webserver_port=41489
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:12.473764 9067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:12.474059 9067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:12.474113 9067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:12.478192 9067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:12.478273 9067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:12.478300 9067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:12.478322 9067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:12.478343 9067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:12.485249 9067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:12.486740 9067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:12.488176 9067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:12.497232 9073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.497200 9072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.497200 9075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:12.497534 9067 server_base.cc:1034] running on GCE node
I20260927 00:53:12.498200 9067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:12.499413 9067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:12.500581 9067 hybrid_clock.cc:648] HybridClock initialized: now 1790470392500555 us; error 60 us; skew 500 ppm
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:12.503448 9067 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:53:12.504390 9067 webserver.cc:460] Webserver started at http://127.0.98.62:41489/ using document root <none> and password file <none>
I20260927 00:53:12.504943 9067 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:12.505025 9067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:12.509133 9067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:53:12.511685 9082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.513180 9067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:53:12.513319 9067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c2bc12ff039a4892bbc9c0e91068566d"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "65cb4c16b4c2da18f4bcf3e8287ab8b6"
server_key_iv: "a5591f7f8a761fb6e6a98f4837bf0212"
server_key_version: "encryptionkey@0"
I20260927 00:53:12.513870 9067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:12.533174 9067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:12.536902 9067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:12.537176 9067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:12.545710 9067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46517
I20260927 00:53:12.545773 9134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46517 every 8 connection(s)
I20260927 00:53:12.547012 9067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:12.551682 9135 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:53:12.553416 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Bootstrap starting.
I20260927 00:53:12.555377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9067
I20260927 00:53:12.555881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45783
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46517
--webserver_port=41629
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:53:12.570253 9135 log.cc:824] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:12.576829 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:53:12.577212 9135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Bootstrap complete.
I20260927 00:53:12.581437 9135 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:12.581866 9135 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bc12ff039a4892bbc9c0e91068566d, State: Initialized, Role: FOLLOWER
I20260927 00:53:12.582362 9135 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:12.582486 9135 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:12.582552 9135 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:12.582651 9135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:53:12.584685 9135 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:12.585006 9135 leader_election.cc:302] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2bc12ff039a4892bbc9c0e91068566d; no voters:
I20260927 00:53:12.585300 9135 leader_election.cc:288] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:53:12.585409 9139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:53:12.585659 9139 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [term 2 LEADER]: Becoming Leader. State: Replica: c2bc12ff039a4892bbc9c0e91068566d, State: Running, Role: LEADER
I20260927 00:53:12.586103 9139 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } }
I20260927 00:53:12.586601 9135 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:12.587654 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c2bc12ff039a4892bbc9c0e91068566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } }
I20260927 00:53:12.587864 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:12.588271 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2bc12ff039a4892bbc9c0e91068566d. Latest consensus state: current_term: 2 leader_uuid: "c2bc12ff039a4892bbc9c0e91068566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc12ff039a4892bbc9c0e91068566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } }
I20260927 00:53:12.588332 9148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:12.588366 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:12.591601 9148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:12.592227 9148 catalog_manager.cc:1259] Loaded cluster ID: cff10f85a4ef4c2f98bc13c9ddffec37
I20260927 00:53:12.592293 9148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:12.594671 9148 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:12.595366 9148 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c2bc12ff039a4892bbc9c0e91068566d: Loaded TSK: 0
I20260927 00:53:12.596630 9148 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:53:12.688849 9137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:12.689215 9137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:12.689283 9137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:12.693862 9137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:12.693949 9137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:12.694039 9137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:12.700207 9137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41629
--tserver_master_addrs=127.0.98.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:12.701632 9137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:12.702947 9137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:53:12.714435 9137 server_base.cc:1034] running on GCE node
W20260927 00:53:12.714460 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.714481 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.714403 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:12.715365 9137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:12.716382 9137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:12.717728 9137 hybrid_clock.cc:648] HybridClock initialized: now 1790470392717670 us; error 92 us; skew 500 ppm
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:12.721634 9137 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:53:12.722941 9137 webserver.cc:460] Webserver started at http://127.0.98.1:41629/ using document root <none> and password file <none>
I20260927 00:53:12.723726 9137 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:12.723836 9137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:12.728922 9137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:53:12.732157 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.734030 9137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:53:12.734202 9137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "c23f0a75b88e4310a28e3d2590a2ecac"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "63a2612e85f8fce473bf67561626420d"
server_key_iv: "93cd87323b97119bfa627a6a95028efe"
server_key_version: "encryptionkey@0"
I20260927 00:53:12.734941 9137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:12.769179 9137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:12.773698 9137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:12.774000 9137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:12.774835 9137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:12.775920 9137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:12.775972 9137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.776023 9137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:12.776053 9137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.787115 9137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45783
I20260927 00:53:12.787216 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45783 every 8 connection(s)
I20260927 00:53:12.788566 9137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:12.794849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9137
I20260927 00:53:12.795544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33267
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46517
--webserver_port=43463
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:12.805611 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:12.791294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:59543)
0927 00:53:12.791901 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:12.791957 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:12.793463 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:12.794935 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:12.794948 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:12.795585 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:12.796445 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.796472 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.798245 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.798248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.799403 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.799416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.800512 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.800516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.800694 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.801503 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:12.801523 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:12.804465 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":154,"threads_started":1}
W20260927 00:53:12.805943 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:12.791261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59543 (local address 127.0.98.62:46517)
0927 00:53:12.791825 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:12.791844 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:12.792104 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:53:12.793950 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:12.793964 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:12.794340 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:12.794693 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:12.796714 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.798040 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.799558 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.800301 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.800483 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.804811 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59543: BlockingRecv error: recv got EOF from 127.0.98.1:59543 (error 108)
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":144,"threads_started":1}
W20260927 00:53:12.806229 9285 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46517: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:12.812270 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:12.806845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:52211)
0927 00:53:12.807032 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:12.807049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:12.807195 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:12.807482 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:12.807485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:12.807669 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:12.807781 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.807787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.808584 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.808588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.809162 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.809171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.809612 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.809616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.809706 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.810245 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:12.810258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:12.812112 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 00:53:12.812440 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:12.806941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52211 (local address 127.0.98.62:46517)
0927 00:53:12.807061 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:12.807066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:12.807109 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:53:12.807312 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:12.807315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:12.807368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:12.807441 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:12.807906 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.808477 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.809273 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.809505 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.809595 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.812319 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52211: BlockingRecv error: recv got EOF from 127.0.98.1:52211 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:12.818440 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:12.813154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:58439)
0927 00:53:12.813361 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:12.813377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:12.813466 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:12.813743 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:12.813745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:12.813949 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:12.814035 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.814041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.814802 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.814805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.815417 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:12.815426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.815837 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.815840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:12.815917 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.816396 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:12.816407 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:12.818286 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260927 00:53:12.818672 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:12.813288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58439 (local address 127.0.98.62:46517)
0927 00:53:12.813447 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:12.813451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:12.813466 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:12.813560 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:12.813563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:12.813609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:12.813715 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:12.814141 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.814688 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.815527 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:12.815736 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:12.815824 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:12.818443 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58439: BlockingRecv error: recv got EOF from 127.0.98.1:58439 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 00:53:12.818774 9285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:53:12.913458 9289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:12.913735 9289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:12.913779 9289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:12.917898 9289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:12.917972 9289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:12.918035 9289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:12.923451 9289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43463
--tserver_master_addrs=127.0.98.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:12.924561 9289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:12.925594 9289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:12.936088 9297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:12.936244 9289 server_base.cc:1034] running on GCE node
W20260927 00:53:12.936101 9294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:12.936101 9295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:12.936996 9289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:12.937810 9289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:12.938951 9289 hybrid_clock.cc:648] HybridClock initialized: now 1790470392938909 us; error 55 us; skew 500 ppm
Sep 27 00:53:12 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470392, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:12.942317 9289 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:53:12.943493 9289 webserver.cc:460] Webserver started at http://127.0.98.2:43463/ using document root <none> and password file <none>
I20260927 00:53:12.944145 9289 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:12.944228 9289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:12.949225 9289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:53:12.952219 9304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.953922 9289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:12.954090 9289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "9cd8cee56a6c42a293f48db66b954971"
format_stamp: "Formatted at 2026-09-27 00:53:11 on dist-test-slave-9fh2"
server_key: "f701dc5ca0b0499a04b94ea3227dc4b4"
server_key_iv: "6fbbfbedd6052c48467e846af691bb48"
server_key_version: "encryptionkey@0"
I20260927 00:53:12.954806 9289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:12.978000 9289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:12.981801 9289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:12.982016 9289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:12.982872 9289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:12.983968 9289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:12.984020 9289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.984081 9289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:12.984107 9289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:12.995589 9289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33267
I20260927 00:53:12.995618 9417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33267 every 8 connection(s)
I20260927 00:53:12.996898 9289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:13.003758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9289
I20260927 00:53:13.004197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34859
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46517
--webserver_port=34763
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:53:13.010772 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:12.999035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47259 (local address 127.0.98.62:46517)
0927 00:53:12.999212 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:12.999217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.000157 (+ 940us) server_negotiation.cc:407] Connection header received
0927 00:53:13.001747 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.001751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.001819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.001930 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.003642 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.004286 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.006192 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.006575 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.006728 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.010564 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47259: BlockingRecv error: recv got EOF from 127.0.98.2:47259 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:53:13.011099 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:12.999382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:47259)
0927 00:53:12.999945 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:12.999998 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.001465 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.002214 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.002223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.002698 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.003434 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.003476 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.004518 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.004524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.005971 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.005989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.006803 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.006807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.006997 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.007758 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.007777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.010209 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":145,"threads_started":1}
W20260927 00:53:13.011969 9418 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46517: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:13.019812 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.012636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:51345)
0927 00:53:13.012870 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:13.012890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.013033 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.013414 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.013417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.013645 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.013833 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.013843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.014697 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.014702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.015739 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.015755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.016337 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.016341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.016466 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.017146 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.017163 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.019617 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 00:53:13.020182 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:13.012807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51345 (local address 127.0.98.62:46517)
0927 00:53:13.012940 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.012944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.012962 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:53:13.013207 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.013210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.013261 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.013345 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.013999 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.014563 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.015899 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.016194 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.016296 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.019817 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51345: BlockingRecv error: recv got EOF from 127.0.98.2:51345 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:13.026772 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.020638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:44571)
0927 00:53:13.020787 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:13.020812 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.020910 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.021217 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.021220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.021405 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.021521 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.021527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.022283 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.022286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.023228 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.023240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.023781 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.023784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.023885 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.024410 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.024425 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.026606 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 00:53:13.026890 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:13.020751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44571 (local address 127.0.98.62:46517)
0927 00:53:13.020865 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.020868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.020881 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:53:13.021017 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.021020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.021063 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.021180 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.021641 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.022187 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.023420 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.023659 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.023743 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.026771 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44571: BlockingRecv error: recv got EOF from 127.0.98.2:44571 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 00:53:13.027302 9418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:53:13.140395 9421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:13.140739 9421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:13.140789 9421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:13.145234 9421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:13.145350 9421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:13.145424 9421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:13.151537 9421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34763
--tserver_master_addrs=127.0.98.62:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:13.153012 9421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:13.154255 9421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:53:13.166518 9421 server_base.cc:1034] running on GCE node
W20260927 00:53:13.166491 9426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:13.166757 9427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:13.166491 9429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:13.167374 9421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:13.168226 9421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:13.169423 9421 hybrid_clock.cc:648] HybridClock initialized: now 1790470393169375 us; error 64 us; skew 500 ppm
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470393, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:13.172715 9421 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:53:13.173882 9421 webserver.cc:460] Webserver started at http://127.0.98.3:34763/ using document root <none> and password file <none>
I20260927 00:53:13.174582 9421 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:13.174674 9421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:13.180159 9421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:53:13.183017 9436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:13.184870 9421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:53:13.185041 9421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "f73e33edfa174c4486e58e0344d075f7"
format_stamp: "Formatted at 2026-09-27 00:53:12 on dist-test-slave-9fh2"
server_key: "1eaf096ccb6ca259cd951b4d18c29d97"
server_key_iv: "59297f13571ff74dd75e607fcb1a043a"
server_key_version: "encryptionkey@0"
I20260927 00:53:13.185771 9421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:13.199663 9421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:13.203518 9421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:13.203728 9421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:13.204460 9421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:13.205484 9421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:13.205536 9421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:13.205595 9421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:13.205622 9421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:13.217263 9421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34859
I20260927 00:53:13.217301 9549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34859 every 8 connection(s)
I20260927 00:53:13.218622 9421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:13.223133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9421
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:53:13.233132 9287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:13.220993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50249 (local address 127.0.98.62:46517)
0927 00:53:13.221173 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.221177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.222352 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 00:53:13.223969 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.223973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.224063 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.224187 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.225870 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.226532 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.227885 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.228252 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.228371 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.232998 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50249: BlockingRecv error: recv got EOF from 127.0.98.3:50249 (error 108)
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":78}
I20260927 00:53:13.233835 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.221496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:50249)
0927 00:53:13.222152 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:13.222208 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.223678 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.224539 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.224547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.225106 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.225655 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.225668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.226692 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.226696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.227751 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.227761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.228445 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.228450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.228646 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.229267 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.229284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.232869 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":169,"threads_started":1}
W20260927 00:53:13.234310 9550 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46517: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 00:53:13.234719 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.224483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.0.98.62:46517)
0927 00:53:13.224779 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.224783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.224799 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:53:13.224845 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.224848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.224900 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.225002 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.226149 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.226805 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.227964 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.228264 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.228415 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.230138 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:13.230144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:13.230160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:13.230187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:13.231968 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:13.232617 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:13.232625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:13.232632 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:13.232841 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:13.233103 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:13.233107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:13.233109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:13.233701 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:13.234000 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:13.234232 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:53:13.234549 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:13.240686 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.234816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:33479)
0927 00:53:13.234994 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:13.235007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.235084 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.235406 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.235408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.235580 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.235700 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.235706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.236598 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.236603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.237287 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.237299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.237737 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.237740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.237841 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.238395 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.238408 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.240522 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260927 00:53:13.240819 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:13.234926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33479 (local address 127.0.98.62:46517)
0927 00:53:13.235037 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.235041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.235056 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:13.235217 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.235220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.235267 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.235370 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.235815 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.236464 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.237401 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.237634 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.237721 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.240692 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33479: BlockingRecv error: recv got EOF from 127.0.98.3:33479 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:13.247134 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.241506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:54471)
0927 00:53:13.241661 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:13.241679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.241798 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.242058 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.242061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.242247 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.242343 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.242348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.243260 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.243263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.243839 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.243848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.244270 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.244273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.244351 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.244959 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.244973 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.246965 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 00:53:13.247534 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:13.241630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54471 (local address 127.0.98.62:46517)
0927 00:53:13.241835 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.241840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.241855 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:13.241902 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.241905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.241947 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.242027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.242460 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.243107 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.243963 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.244174 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.244257 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.247121 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54471: BlockingRecv error: recv got EOF from 127.0.98.3:54471 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260927 00:53:13.247529 9550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:13 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:13.825460 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:13.819615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:36629)
0927 00:53:13.819971 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:13.819989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:13.820194 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:13.820474 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:13.820477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:13.820680 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:13.820801 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.820807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.821548 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.821551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.822109 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:13.822117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.822529 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.822532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:13.822622 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.823257 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:13.823270 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:13.825282 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
W20260927 00:53:13.825564 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:13.819692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36629 (local address 127.0.98.62:46517)
0927 00:53:13.820020 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:13.820024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:13.820173 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:53:13.820306 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:13.820309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:13.820356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:13.820432 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:13.820914 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.821440 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.822217 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:13.822429 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:13.822520 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:13.825442 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36629: BlockingRecv error: recv got EOF from 127.0.98.1:36629 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":119,"threads_started":1}
Sep 27 00:53:14 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:14 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:14.033555 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:14.027928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:45081)
0927 00:53:14.028183 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:14.028199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:14.028320 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:14.028555 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:14.028558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:14.028754 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:14.028856 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:14.028862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.029607 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.029611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:14.030193 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:14.030201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.030655 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.030659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:14.030739 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:14.031319 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:14.031333 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:14.033394 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
W20260927 00:53:14.033864 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:14.028134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45081 (local address 127.0.98.62:46517)
0927 00:53:14.028293 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:14.028297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:14.028311 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:53:14.028410 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:14.028413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:14.028457 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:14.028518 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:14.028966 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.029495 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.030303 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.030554 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.030638 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:14.033664 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45081: BlockingRecv error: recv got EOF from 127.0.98.2:45081 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 00:53:14 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:14 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:14.254971 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:14.248429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:56855)
0927 00:53:14.248834 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:14.248853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:14.249002 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:14.249347 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:14.249349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:14.249546 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:14.249680 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:14.249686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.250424 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.250427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:14.251004 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:14.251013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.251501 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.251505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:14.251600 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:14.252276 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:14.252288 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:14.254796 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":98,"threads_started":1}
W20260927 00:53:14.255280 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:14.248627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56855 (local address 127.0.98.62:46517)
0927 00:53:14.248792 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:14.248797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:14.248953 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:53:14.249171 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:14.249174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:14.249231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:14.249300 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:14.249799 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.250313 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.251121 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.251387 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.251477 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:14.255089 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56855: BlockingRecv error: recv got EOF from 127.0.98.3:56855 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 00:53:14 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:14 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:14.832403 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:14.826519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:43623)
0927 00:53:14.826786 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:14.826800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:14.826896 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:14.827258 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:14.827260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:14.827475 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:14.827588 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:14.827595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.828471 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.828474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:14.829028 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:14.829035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.829466 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.829470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:14.829544 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:14.830098 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:14.830109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:14.832224 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
W20260927 00:53:14.832536 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:14.826646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43623 (local address 127.0.98.62:46517)
0927 00:53:14.826944 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:14.826948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:14.826966 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:53:14.827013 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:14.827015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:14.827081 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:14.827243 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:14.827734 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.828358 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.829140 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:14.829366 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:14.829451 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:14.832384 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43623: BlockingRecv error: recv got EOF from 127.0.98.1:43623 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
Sep 27 00:53:15 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:15 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:15.040951 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:15.034478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:43247)
0927 00:53:15.034815 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:15.034833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:15.034960 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:15.035232 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:15.035235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:15.035448 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:15.035574 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:15.035579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.036434 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.036438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:15.037092 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:15.037103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.037596 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.037599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:15.037676 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:15.038313 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:15.038327 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:15.040739 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
W20260927 00:53:15.041000 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:15.034572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43247 (local address 127.0.98.62:46517)
0927 00:53:15.034750 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:15.034754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:15.034900 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:53:15.035037 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:15.035040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:15.035088 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:15.035211 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:15.035708 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.036260 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.037240 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.037464 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.037555 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:15.040891 (+ 3336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43247: BlockingRecv error: recv got EOF from 127.0.98.2:43247 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 00:53:15 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:15 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:15.263293 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:15.256129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:51323)
0927 00:53:15.256472 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:15.256541 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:15.256712 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:15.257045 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:15.257047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:15.257260 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:15.257395 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:15.257401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.258493 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.258497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:15.259279 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:15.259289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.259982 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.259986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:15.260083 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:15.260719 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:15.260732 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:15.263045 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
W20260927 00:53:15.263468 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:15.256266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51323 (local address 127.0.98.62:46517)
0927 00:53:15.256404 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:15.256410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:15.256647 (+ 237us) server_negotiation.cc:407] Connection header received
0927 00:53:15.256818 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:15.256821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:15.256887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:15.257023 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:15.257516 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.258348 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.259420 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.259772 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.259973 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:15.263319 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51323: BlockingRecv error: recv got EOF from 127.0.98.3:51323 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 00:53:15 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:15 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:15.839797 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:15.833359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:60169)
0927 00:53:15.833734 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:15.833777 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:15.833917 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:15.834223 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:15.834226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:15.834487 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:15.834610 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:15.834618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.835514 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.835518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:15.836148 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:15.836159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.836633 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.836637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:15.836727 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:15.837366 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:15.837379 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:15.839623 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":78,"threads_started":1}
W20260927 00:53:15.839990 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:15.833452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60169 (local address 127.0.98.62:46517)
0927 00:53:15.833836 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:15.833842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:15.833858 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:53:15.834027 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:15.834031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:15.834092 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:15.834181 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:15.834734 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.835393 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.836289 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:15.836520 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:15.836616 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:15.839826 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60169: BlockingRecv error: recv got EOF from 127.0.98.1:60169 (error 108)
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":158,"threads_started":1}
Sep 27 00:53:16 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:16 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:16.047957 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:16.041842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:55853)
0927 00:53:16.042201 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:16.042218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:16.042343 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:16.042608 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:16.042610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:16.042792 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:16.042909 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:16.042915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.043865 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.043868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:16.044459 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:16.044469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.045002 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.045006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:16.045094 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:16.045632 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:16.045643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:16.047797 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
W20260927 00:53:16.048182 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:16.041963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55853 (local address 127.0.98.62:46517)
0927 00:53:16.042160 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:16.042163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:16.042301 (+ 138us) server_negotiation.cc:407] Connection header received
0927 00:53:16.042443 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:16.042445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:16.042501 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:16.042571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:16.043022 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.043719 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.044601 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.044856 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.044952 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:16.048015 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55853: BlockingRecv error: recv got EOF from 127.0.98.2:55853 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 00:53:16 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:16 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:16.270411 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:16.264621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:38401)
0927 00:53:16.264990 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:16.265005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:16.265149 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:16.265412 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:16.265414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:16.265615 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:16.265716 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:16.265722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.266487 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.266490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:16.267068 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:16.267075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.267528 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.267531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:16.267607 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:16.268107 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:16.268119 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:16.270247 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
W20260927 00:53:16.270506 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:16.264740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38401 (local address 127.0.98.62:46517)
0927 00:53:16.264936 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:16.264941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:16.265099 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:53:16.265245 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:16.265249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:16.265300 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:16.265370 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:16.265834 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.266380 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.267186 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.267420 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.267508 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:16.270396 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38401: BlockingRecv error: recv got EOF from 127.0.98.3:38401 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 00:53:16 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:16 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:16.850396 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:16.840878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:59935)
0927 00:53:16.841211 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:16.841227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:16.841360 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:16.841621 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:16.841623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:16.841820 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:16.841924 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:16.841931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.842695 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.842699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:16.843290 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:16.843298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.843727 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.843731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:16.843818 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:16.847931 (+ 4113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:16.847945 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:16.850246 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
W20260927 00:53:16.850786 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:16.840895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59935 (local address 127.0.98.62:46517)
0927 00:53:16.841203 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:16.841207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:16.841381 (+ 174us) server_negotiation.cc:407] Connection header received
0927 00:53:16.841450 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:16.841453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:16.841501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:16.841581 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:16.842033 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.842582 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.843395 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:16.843629 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:16.843710 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:16.850601 (+ 6891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59935: BlockingRecv error: recv got EOF from 127.0.98.1:59935 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
Sep 27 00:53:17 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:17 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:17.055135 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:17.048840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:46025)
0927 00:53:17.049294 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:17.049313 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:17.049446 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:17.049791 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:17.049794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:17.049983 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:17.050161 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:17.050169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.051096 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.051099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:17.051761 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:17.051770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.052264 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.052266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:17.052326 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:17.052881 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:17.052892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:17.054947 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":166,"threads_started":1}
W20260927 00:53:17.055266 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:17.048992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46025 (local address 127.0.98.62:46517)
0927 00:53:17.049188 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:17.049195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:17.049419 (+ 224us) server_negotiation.cc:407] Connection header received
0927 00:53:17.049555 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:17.049557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:17.049621 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:17.049708 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:17.050376 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.050968 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.051906 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.052151 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.052257 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:17.055101 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46025: BlockingRecv error: recv got EOF from 127.0.98.2:46025 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 00:53:17 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:17 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:17.277552 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:17.271561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:48605)
0927 00:53:17.271931 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:17.271948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:17.272074 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:17.272327 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:17.272329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:17.272521 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:17.272607 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:17.272612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.273378 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.273382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:17.273910 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:17.273917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.274327 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.274330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:17.274404 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:17.274897 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:17.274908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:17.277362 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":155,"threads_started":1}
W20260927 00:53:17.277650 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:17.271694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48605 (local address 127.0.98.62:46517)
0927 00:53:17.271893 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:17.271898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:17.272028 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:53:17.272165 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:17.272168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:17.272220 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:17.272290 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:17.272736 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.273276 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.274016 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.274223 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.274303 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:17.277518 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48605: BlockingRecv error: recv got EOF from 127.0.98.3:48605 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 00:53:17 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:17 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:17.857518 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:17.851451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:43609)
0927 00:53:17.851810 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:17.851827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:17.851953 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:17.852222 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:17.852224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:17.852495 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:17.852619 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:17.852626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.853462 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.853465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:17.854095 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:17.854104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.854523 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.854526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:17.854594 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:17.855206 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:17.855220 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:17.857345 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
W20260927 00:53:17.857671 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:17.851563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43609 (local address 127.0.98.62:46517)
0927 00:53:17.851853 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:17.851858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:17.851905 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:53:17.852043 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:17.852046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:17.852102 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:17.852189 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:17.852743 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.853285 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.854208 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:17.854423 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:17.854498 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:17.857533 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43609: BlockingRecv error: recv got EOF from 127.0.98.1:43609 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
Sep 27 00:53:18 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:18 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:18.062616 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:18.056171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:41071)
0927 00:53:18.056552 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:18.056567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:18.056696 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:18.056955 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:18.056957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:18.057178 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:18.057299 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:18.057304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.058084 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.058087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:18.058861 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:18.058872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.059527 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.059530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:18.059610 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:18.060200 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:18.060213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:18.062349 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
W20260927 00:53:18.062681 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:18.056335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41071 (local address 127.0.98.62:46517)
0927 00:53:18.056459 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:18.056463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:18.056683 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:53:18.056778 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:18.056781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:18.056827 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:18.056914 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:18.057413 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.057946 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.059017 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.059349 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.059444 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:18.062563 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41071: BlockingRecv error: recv got EOF from 127.0.98.2:41071 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 00:53:18 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:18 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:18.284762 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:18.278591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:38561)
0927 00:53:18.278949 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:18.278964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:18.279095 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:18.279509 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:18.279512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:18.279726 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:18.279854 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:18.279860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.280648 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.280651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:18.281299 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:18.281307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.281802 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.281805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:18.281872 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:18.282509 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:18.282522 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:18.284605 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
W20260927 00:53:18.285072 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:18.278700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38561 (local address 127.0.98.62:46517)
0927 00:53:18.278823 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:18.278826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:18.279051 (+ 225us) server_negotiation.cc:407] Connection header received
0927 00:53:18.279229 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:18.279231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:18.279279 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:18.279351 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:18.279993 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.280520 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.281461 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.281678 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.281778 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:18.284899 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38561: BlockingRecv error: recv got EOF from 127.0.98.3:38561 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 00:53:18 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:18 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:18.865487 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:18.858585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:54141)
0927 00:53:18.858937 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:18.858959 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:18.859118 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:18.859436 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:18.859438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:18.859642 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:18.859783 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:18.859791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.860674 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.860677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:18.861606 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:18.861620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.862160 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.862164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:18.862308 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:18.862852 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:18.862864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:18.865337 (+ 2473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
W20260927 00:53:18.865765 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:18.858802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54141 (local address 127.0.98.62:46517)
0927 00:53:18.859191 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:18.859195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:18.859213 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:53:18.859259 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:18.859261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:18.859306 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:18.859393 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:18.859921 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.860511 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.861747 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:18.862022 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:18.862114 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:18.865502 (+ 3388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54141: BlockingRecv error: recv got EOF from 127.0.98.1:54141 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
Sep 27 00:53:19 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:19 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:19.069977 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:19.063763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:56541)
0927 00:53:19.064150 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:19.064170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:19.064314 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:19.064591 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:19.064595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:19.064783 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:19.064885 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:19.064891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.065617 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.065621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:19.066181 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:19.066188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.066600 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.066604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:19.066681 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:19.067330 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:19.067345 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:19.069842 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":111,"threads_started":1}
W20260927 00:53:19.070415 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:19.063935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56541 (local address 127.0.98.62:46517)
0927 00:53:19.064168 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:19.064173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:19.064257 (+ 84us) server_negotiation.cc:407] Connection header received
0927 00:53:19.064405 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:19.064408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:19.064452 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:19.064519 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:19.064999 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.065514 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.066289 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.066495 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.066578 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:19.070051 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56541: BlockingRecv error: recv got EOF from 127.0.98.2:56541 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 27 00:53:19 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:19 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:19.292676 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:19.286056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:45835)
0927 00:53:19.286394 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:19.286414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:19.286577 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:19.286860 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:19.286863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:19.287100 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:19.287248 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:19.287255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.288363 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.288367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:19.289120 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:19.289129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.289592 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.289595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:19.289674 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:19.290303 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:19.290316 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:19.292484 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
W20260927 00:53:19.292781 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:19.286230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45835 (local address 127.0.98.62:46517)
0927 00:53:19.286390 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:19.286393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:19.286552 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:53:19.286689 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:19.286693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:19.286748 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:19.286830 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:19.287389 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.288204 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.289228 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.289480 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.289569 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:19.292664 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45835: BlockingRecv error: recv got EOF from 127.0.98.3:45835 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:53:19 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:19 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:19.872994 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:19.866415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:55539)
0927 00:53:19.866813 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:19.866830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:19.866988 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:19.867336 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:19.867338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:19.867630 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:19.867780 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:19.867789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.868627 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.868630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:19.869249 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:19.869259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.869735 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.869738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:19.869841 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:19.870528 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:19.870540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:19.872820 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":81,"threads_started":1}
W20260927 00:53:19.873085 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:19.866636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55539 (local address 127.0.98.62:46517)
0927 00:53:19.866893 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:19.866897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:19.866919 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:53:19.867113 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:19.867116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:19.867197 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:19.867283 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:19.867909 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.868476 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.869373 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:19.869598 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:19.869680 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:19.872963 (+ 3283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55539: BlockingRecv error: recv got EOF from 127.0.98.1:55539 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
Sep 27 00:53:20 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:20 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:20.076678 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:20.070891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:55499)
0927 00:53:20.071275 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:20.071292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:20.071442 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:20.071742 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:20.071745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:20.071968 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:20.072094 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:20.072100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.072910 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.072913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:20.073520 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:20.073529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.073938 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.073941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:20.074005 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:20.074539 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:20.074551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:20.076533 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
W20260927 00:53:20.076794 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:20.071047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55499 (local address 127.0.98.62:46517)
0927 00:53:20.071276 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:20.071281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:20.071415 (+ 134us) server_negotiation.cc:407] Connection header received
0927 00:53:20.071525 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:20.071527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:20.071577 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:20.071653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:20.072216 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.072762 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.073617 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.073844 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.073919 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:20.076676 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55499: BlockingRecv error: recv got EOF from 127.0.98.2:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 27 00:53:20 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:20 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:20.300132 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:20.293710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:57353)
0927 00:53:20.294134 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:20.294150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:20.294306 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:20.294654 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:20.294657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:20.294937 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:20.295055 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:20.295062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.295961 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.295964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:20.296598 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:20.296607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.297098 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.297101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:20.297181 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:20.297813 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:20.297826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:20.299966 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":106,"threads_started":1}
W20260927 00:53:20.300333 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:20.293832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57353 (local address 127.0.98.62:46517)
0927 00:53:20.294055 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:20.294060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:20.294249 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:53:20.294478 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:20.294480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:20.294532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:20.294608 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:20.295219 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.295846 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.296721 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.296971 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.297073 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:20.300197 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57353: BlockingRecv error: recv got EOF from 127.0.98.3:57353 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 00:53:20 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:20 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:20.879675 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:20.874097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:36671)
0927 00:53:20.874482 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:20.874501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:20.874633 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:20.875039 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:20.875043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:20.875281 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:20.875395 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:20.875402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.876176 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.876180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:20.876730 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:20.876738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.877206 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.877210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:20.877287 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:20.877790 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:20.877801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:20.879519 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":127,"threads_started":1}
W20260927 00:53:20.879797 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:20.874392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36671 (local address 127.0.98.62:46517)
0927 00:53:20.874724 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:20.874729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:20.874750 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:53:20.874802 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:20.874804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:20.874852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:20.874950 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:20.875532 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.876069 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.876845 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:20.877089 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:20.877179 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:20.879680 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36671: BlockingRecv error: recv got EOF from 127.0.98.1:36671 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":117,"threads_started":1}
Sep 27 00:53:21 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:21 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:21.084295 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:21.077737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:33007)
0927 00:53:21.078139 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:21.078158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:21.078332 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:21.078616 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:21.078619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:21.078858 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:21.078970 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:21.078976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.080093 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.080096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:21.080797 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:21.080806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.081248 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.081251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:21.081334 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:21.081924 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:21.081936 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:21.084101 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
W20260927 00:53:21.084403 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:21.077954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33007 (local address 127.0.98.62:46517)
0927 00:53:21.078141 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:21.078146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:21.078263 (+ 117us) server_negotiation.cc:407] Connection header received
0927 00:53:21.078450 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:21.078454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:21.078503 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:21.078575 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:21.079098 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.079984 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.080904 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.081135 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.081219 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:21.084286 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33007: BlockingRecv error: recv got EOF from 127.0.98.2:33007 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 00:53:21 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:21 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:21.307082 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:21.301201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:43635)
0927 00:53:21.301537 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:21.301557 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:21.301692 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:21.301964 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:21.301966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:21.302151 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:21.302256 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:21.302262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.303084 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.303087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:21.303762 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:21.303770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.304226 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.304229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:21.304289 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:21.304835 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:21.304845 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:21.306896 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
W20260927 00:53:21.307201 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:21.301283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43635 (local address 127.0.98.62:46517)
0927 00:53:21.301516 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:21.301520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:21.301640 (+ 120us) server_negotiation.cc:407] Connection header received
0927 00:53:21.301793 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:21.301796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:21.301850 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:21.301923 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:21.302396 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.302983 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.303880 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.304121 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.304211 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:21.307054 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43635: BlockingRecv error: recv got EOF from 127.0.98.3:43635 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:53:21 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:21 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:21.891727 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:21.880768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:55287)
0927 00:53:21.881113 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:21.881130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:21.881261 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:21.881556 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:21.881558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:21.881787 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:21.881915 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:21.881921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.882715 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.882718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:21.883487 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:21.883500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.884013 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.884017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:21.884125 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:21.889081 (+ 4956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:21.889096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:21.891543 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
W20260927 00:53:21.891878 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:21.880919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55287 (local address 127.0.98.62:46517)
0927 00:53:21.881209 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:21.881213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:21.881230 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:53:21.881360 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:21.881362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:21.881410 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:21.881496 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:21.882020 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.882594 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.883599 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:21.883880 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:21.883972 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:21.891718 (+ 7746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55287: BlockingRecv error: recv got EOF from 127.0.98.1:55287 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
Sep 27 00:53:22 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:22 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:22.091897 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:22.085417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:53993)
0927 00:53:22.085840 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:22.085861 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:22.086018 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:22.086281 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:22.086284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:22.086549 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:22.086655 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:22.086661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.087514 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.087518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:22.088180 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:22.088190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.088687 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.088692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:22.088829 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:22.089550 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:22.089562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:22.091743 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":122,"threads_started":1}
W20260927 00:53:22.092033 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:22.085602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53993 (local address 127.0.98.62:46517)
0927 00:53:22.085820 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:22.085825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:22.085955 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:53:22.086115 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:22.086118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:22.086165 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:22.086241 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:22.086775 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.087418 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.088283 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.088540 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.088645 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:22.091915 (+ 3270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53993: BlockingRecv error: recv got EOF from 127.0.98.2:53993 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 00:53:22 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:22 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:22.314203 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:22.308169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:54675)
0927 00:53:22.308502 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:22.308517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:22.308675 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:22.308964 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:22.308967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:22.309192 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:22.309335 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:22.309344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.310277 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.310281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:22.311052 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:22.311063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.311545 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.311548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:22.311616 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:22.312142 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:22.312155 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:22.314061 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
W20260927 00:53:22.314381 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:22.308331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54675 (local address 127.0.98.62:46517)
0927 00:53:22.308488 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:22.308492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:22.308641 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:53:22.308768 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:22.308771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:22.308827 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:22.308911 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:22.309483 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.310092 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.311212 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.311433 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.311513 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:22.314242 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54675: BlockingRecv error: recv got EOF from 127.0.98.3:54675 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:53:22 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:22 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:22.898993 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:22.892722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:42825)
0927 00:53:22.893051 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:22.893069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:22.893207 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:22.893521 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:22.893525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:22.893744 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:22.893874 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:22.893880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.894684 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.894687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:22.895474 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:22.895491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.896044 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.896047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:22.896124 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:22.896830 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:22.896846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:22.898833 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":89,"threads_started":1}
W20260927 00:53:22.899108 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:22.892854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42825 (local address 127.0.98.62:46517)
0927 00:53:22.893092 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:22.893095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:22.893171 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:53:22.893299 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:22.893302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:22.893348 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:22.893434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:22.894013 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.894567 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.895631 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:22.895908 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:22.896032 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:22.898972 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42825: BlockingRecv error: recv got EOF from 127.0.98.1:42825 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":60,"threads_started":1}
Sep 27 00:53:23 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:23 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:23.099016 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:23.092956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:42607)
0927 00:53:23.093277 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:23.093294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:23.093436 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:23.093768 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:23.093771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:23.094062 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:23.094199 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:23.094207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.094987 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.094990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:23.095731 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:23.095741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.096306 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.096311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:23.096384 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:23.096927 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:23.096939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:23.098871 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":118,"threads_started":1}
W20260927 00:53:23.099287 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:23.093047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42607 (local address 127.0.98.62:46517)
0927 00:53:23.093200 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:23.093205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:23.093387 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:53:23.093553 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:23.093557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:23.093615 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:23.093705 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:23.094331 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.094869 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.095883 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.096157 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.096257 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:23.099057 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42607: BlockingRecv error: recv got EOF from 127.0.98.2:42607 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 00:53:23 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:23 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:23.321291 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:23.315292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:49823)
0927 00:53:23.315609 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:23.315627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:23.315755 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:23.316050 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:23.316054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:23.316262 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:23.316369 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:23.316377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.317284 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.317286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:23.317891 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:23.317899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.318392 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.318396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:23.318479 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:23.319051 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:23.319065 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:23.321163 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":81,"threads_started":1}
W20260927 00:53:23.321597 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:23.315419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49823 (local address 127.0.98.62:46517)
0927 00:53:23.315595 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:23.315599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:23.315730 (+ 131us) server_negotiation.cc:407] Connection header received
0927 00:53:23.315865 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:23.315867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:23.315916 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:23.316003 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:23.316490 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.317167 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.318000 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.318281 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.318376 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:23.321440 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49823: BlockingRecv error: recv got EOF from 127.0.98.3:49823 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 00:53:23 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:23 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:23.906152 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:23.900027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:45617)
0927 00:53:23.900354 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:23.900369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:23.900467 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:23.900854 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:23.900857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:23.901040 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:23.901208 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:23.901215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.902117 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.902120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:23.902739 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:23.902748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.903232 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.903236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:23.903306 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:23.903871 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:23.903882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:23.905982 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
W20260927 00:53:23.906363 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:23.900135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45617 (local address 127.0.98.62:46517)
0927 00:53:23.900439 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:23.900443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:23.900460 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:53:23.900639 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:23.900643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:23.900705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:23.900811 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:23.901373 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.901980 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.902843 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:23.903092 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:23.903216 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:23.906194 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45617: BlockingRecv error: recv got EOF from 127.0.98.1:45617 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":72,"threads_started":1}
Sep 27 00:53:24 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:24 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:24.105785 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:24.100105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:46357)
0927 00:53:24.100521 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:24.100538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:24.100679 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:24.100944 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:24.100946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:24.101164 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:24.101284 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:24.101290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.102060 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.102063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:24.102668 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:24.102675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.103078 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.103080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:24.103140 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:24.103764 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:24.103776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:24.105612 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":129,"threads_started":1}
W20260927 00:53:24.105885 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:24.100237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46357 (local address 127.0.98.62:46517)
0927 00:53:24.100434 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:24.100438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:24.100635 (+ 197us) server_negotiation.cc:407] Connection header received
0927 00:53:24.100774 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:24.100776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:24.100827 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:24.100909 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:24.101411 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.101961 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.102764 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.102976 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.103062 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:24.105768 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46357: BlockingRecv error: recv got EOF from 127.0.98.2:46357 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 00:53:24 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:24 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:24.328478 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:24.322126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:57823)
0927 00:53:24.322492 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:24.322510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:24.322642 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:24.322903 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:24.322905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:24.323128 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:24.323285 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:24.323291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.324101 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.324104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:24.324781 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:24.324792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.325263 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.325266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:24.325345 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:24.326083 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:24.326099 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:24.328318 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":91,"threads_started":1}
W20260927 00:53:24.328580 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:24.322233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57823 (local address 127.0.98.62:46517)
0927 00:53:24.322379 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:24.322384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:24.322588 (+ 204us) server_negotiation.cc:407] Connection header received
0927 00:53:24.322734 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:24.322736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:24.322786 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:24.322858 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:24.323459 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.323990 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.324897 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.325115 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.325197 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:24.328466 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57823: BlockingRecv error: recv got EOF from 127.0.98.3:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 00:53:24 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:24 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:24.913515 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:24.907333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:58121)
0927 00:53:24.907652 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:24.907680 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:24.907816 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:24.908134 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:24.908136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:24.908464 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:24.908597 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:24.908604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.909517 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.909520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:24.910131 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:24.910139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.910619 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.910622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:24.910699 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:24.911432 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:24.911443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:24.913372 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
W20260927 00:53:24.913659 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:24.907505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58121 (local address 127.0.98.62:46517)
0927 00:53:24.907850 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:24.907854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:24.907871 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:53:24.907917 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:24.907919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:24.907967 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:24.908056 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:24.908774 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.909382 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.910266 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:24.910497 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:24.910588 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:24.913530 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58121: BlockingRecv error: recv got EOF from 127.0.98.1:58121 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
Sep 27 00:53:25 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:25 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:25.117769 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:25.106855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:41513)
0927 00:53:25.107223 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:25.107239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:25.107360 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:25.107687 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:25.107690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:25.107895 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:25.108023 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:25.108029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.108812 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.108814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:25.109414 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:25.109421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.109846 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.109848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:25.109907 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:25.115207 (+ 5300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:25.115222 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:25.117605 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
W20260927 00:53:25.117928 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:25.107029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41513 (local address 127.0.98.62:46517)
0927 00:53:25.107258 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:25.107261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:25.107289 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:53:25.107459 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:25.107461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:25.107506 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:25.107596 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:25.108182 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.108712 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.109527 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.109751 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.109833 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:25.117801 (+ 7968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41513: BlockingRecv error: recv got EOF from 127.0.98.2:41513 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 00:53:25 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:25 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:25.335839 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:25.329513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:37389)
0927 00:53:25.329851 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:25.329871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:25.330042 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:25.330431 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:25.330434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:25.330655 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:25.330773 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:25.330778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.331639 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.331643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:25.332252 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:25.332261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.332693 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.332696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:25.332774 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:25.333421 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:25.333433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:25.335649 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
W20260927 00:53:25.335988 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:25.329674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37389 (local address 127.0.98.62:46517)
0927 00:53:25.329870 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:25.329876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:25.329973 (+ 97us) server_negotiation.cc:407] Connection header received
0927 00:53:25.330147 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:25.330151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:25.330227 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:25.330307 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:25.330909 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.331534 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.332363 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.332588 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.332675 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:25.335827 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37389: BlockingRecv error: recv got EOF from 127.0.98.3:37389 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 00:53:25 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:25 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:25.920859 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:25.914460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:41707)
0927 00:53:25.914774 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:25.914790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:25.914921 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:25.915400 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:25.915403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:25.915623 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:25.915748 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:25.915755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.916601 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.916604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:25.917245 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:25.917258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.917773 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.917776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:25.917858 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:25.918512 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:25.918525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:25.920693 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
W20260927 00:53:25.920972 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:25.914583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41707 (local address 127.0.98.62:46517)
0927 00:53:25.914811 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:25.914815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:25.914878 (+ 63us) server_negotiation.cc:407] Connection header received
0927 00:53:25.915064 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:25.915067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:25.915197 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:25.915300 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:25.915866 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.916440 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.917377 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:25.917626 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:25.917717 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:25.920853 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41707: BlockingRecv error: recv got EOF from 127.0.98.1:41707 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":57,"threads_started":1}
Sep 27 00:53:26 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:26 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:26.124817 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:26.118723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:48307)
0927 00:53:26.118966 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:26.118980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:26.119086 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:26.119449 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:26.119453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:26.119745 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:26.119931 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:26.119943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.120946 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.120950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:26.121587 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:26.121595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.122006 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.122009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:26.122075 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:26.122607 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:26.122619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:26.124636 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
W20260927 00:53:26.124884 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:26.118946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48307 (local address 127.0.98.62:46517)
0927 00:53:26.119116 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:26.119121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:26.119136 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:53:26.119209 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:26.119212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:26.119259 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:26.119339 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:26.120115 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.120819 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.121696 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.121907 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.121986 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:26.124784 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48307: BlockingRecv error: recv got EOF from 127.0.98.2:48307 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 00:53:26 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:26 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:26.343111 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:26.336911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:46867)
0927 00:53:26.337347 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:26.337366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:26.337514 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:26.337820 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:26.337823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:26.338047 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:26.338180 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:26.338186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.339105 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.339109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:26.339932 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:26.339944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.340463 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.340467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:26.340547 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:26.341152 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:26.341169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:26.342952 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":147,"threads_started":1}
W20260927 00:53:26.343235 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:26.336977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46867 (local address 127.0.98.62:46517)
0927 00:53:26.337166 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:26.337170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:26.337523 (+ 353us) server_negotiation.cc:407] Connection header received
0927 00:53:26.337596 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:26.337598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:26.337651 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:26.337750 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:26.338355 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.338916 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.340102 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.340345 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.340434 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:26.343100 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46867: BlockingRecv error: recv got EOF from 127.0.98.3:46867 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 00:53:26 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:26 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:26.931789 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:26.921889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:59559)
0927 00:53:26.922234 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:26.922249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:26.922379 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:26.922770 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:26.922773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:26.922985 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:26.923113 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:26.923119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.924062 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.924065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:26.924695 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:26.924715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.925142 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.925145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:26.925226 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:26.929338 (+ 4112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:26.929351 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:26.931593 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
W20260927 00:53:26.931866 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:26.921951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59559 (local address 127.0.98.62:46517)
0927 00:53:26.922273 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:26.922277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:26.922332 (+ 55us) server_negotiation.cc:407] Connection header received
0927 00:53:26.922519 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:26.922522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:26.922585 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:26.922690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:26.923317 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.923919 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.924813 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:26.925033 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:26.925117 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:26.931751 (+ 6634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59559: BlockingRecv error: recv got EOF from 127.0.98.1:59559 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
Sep 27 00:53:27 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:27 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:27.131690 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:27.125937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:39039)
0927 00:53:27.126322 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:27.126337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:27.126475 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:27.126750 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:27.126752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:27.126949 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:27.127045 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:27.127050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.127809 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.127812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:27.128371 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:27.128379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.128795 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.128799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:27.128872 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:27.129399 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:27.129411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:27.131506 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
W20260927 00:53:27.131773 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:27.126043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39039 (local address 127.0.98.62:46517)
0927 00:53:27.126220 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:27.126225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:27.126426 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:53:27.126569 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:27.126572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:27.126617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:27.126701 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:27.127176 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.127704 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.128477 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.128692 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.128777 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:27.131673 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39039: BlockingRecv error: recv got EOF from 127.0.98.2:39039 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:53:27 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:27 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:27.350872 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:27.344157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.3:38441)
0927 00:53:27.344584 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:27.344604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:27.344803 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:27.345176 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:27.345179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:27.345447 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:27.345607 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:27.345615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.346581 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.346584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:27.347281 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:27.347291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.347716 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.347718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:27.347787 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:27.348415 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:27.348509 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:27.350706 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":115,"threads_started":1}
W20260927 00:53:27.351027 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:27.344352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38441 (local address 127.0.98.62:46517)
0927 00:53:27.344503 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:27.344508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:27.344729 (+ 221us) server_negotiation.cc:407] Connection header received
0927 00:53:27.344979 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:27.344982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:27.345039 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:27.345124 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:27.345736 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.346461 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.347389 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.347607 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.347697 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:27.350899 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38441: BlockingRecv error: recv got EOF from 127.0.98.3:38441 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:53:27 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:27 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:27.938767 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:27.932954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.1:36883)
0927 00:53:27.933290 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:27.933306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:27.933422 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:27.933706 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:27.933709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:27.933900 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:27.934006 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:27.934012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.934880 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.934883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:27.935548 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:27.935558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.936146 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.936149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:27.936223 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:27.936771 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:27.936782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:27.938582 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
W20260927 00:53:27.938967 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:27.933020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36883 (local address 127.0.98.62:46517)
0927 00:53:27.933326 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:27.933331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:27.933386 (+ 55us) server_negotiation.cc:407] Connection header received
0927 00:53:27.933517 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:27.933519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:27.933572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:27.933667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:27.934123 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.934748 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.935684 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:27.935937 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:27.936041 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:27.938804 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36883: BlockingRecv error: recv got EOF from 127.0.98.1:36883 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
Sep 27 00:53:28 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:53:28 dist-test-slave-9fh2 krb5kdc[8554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:53:28.138681 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:28.132909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.98.2:58907)
0927 00:53:28.133463 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:28.133479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:28.133606 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:28.133878 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:28.133881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:28.134094 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:28.134206 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:28.134213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.135027 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.135031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:28.135704 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:28.135713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.136140 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.136143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:28.136213 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.136769 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:28.136780 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:28.138460 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":93,"threads_started":1}
W20260927 00:53:28.138716 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:28.133035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58907 (local address 127.0.98.62:46517)
0927 00:53:28.133232 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:28.133237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:28.133582 (+ 345us) server_negotiation.cc:407] Connection header received
0927 00:53:28.133724 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:28.133727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:28.133776 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:28.133845 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:28.134315 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.134927 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.135817 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.136033 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.136130 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.138605 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58907: BlockingRecv error: recv got EOF from 127.0.98.2:58907 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:53:28.229151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9137
I20260927 00:53:28.235204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9289
I20260927 00:53:28.240615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9421
I20260927 00:53:28.246096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9067
2026-09-27T00:53:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18136 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T00:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:28Z Disabled control of system clock
I20260927 00:53:28.278690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38817
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:53:28.408864 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:28.409212 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:28.409250 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:28.413580 9617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:28.413671 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:28.413684 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:28.413700 9617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:28.413715 9617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:28.419289 9617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:28.420866 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:28.422657 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:28.433248 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:28.433314 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:28.433507 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:28.433569 9617 server_base.cc:1034] running on GCE node
I20260927 00:53:28.435823 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:28.437080 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:28.438256 9617 hybrid_clock.cc:648] HybridClock initialized: now 1790470408438228 us; error 58 us; skew 500 ppm
I20260927 00:53:28.440402 9617 webserver.cc:460] Webserver started at http://127.0.98.62:34361/ using document root <none> and password file <none>
I20260927 00:53:28.441035 9617 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:28.441112 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:28.441344 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:28.443557 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "37bc51a7a0344992bab0582befbe1c2f"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "f525204970ca20e02fb898360526f9ba"
server_key_iv: "01820509013a1b3cec4076961db68ad7"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.444121 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "37bc51a7a0344992bab0582befbe1c2f"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "f525204970ca20e02fb898360526f9ba"
server_key_iv: "01820509013a1b3cec4076961db68ad7"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.448312 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:53:28.451184 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.452870 9617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:28.453013 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "37bc51a7a0344992bab0582befbe1c2f"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "f525204970ca20e02fb898360526f9ba"
server_key_iv: "01820509013a1b3cec4076961db68ad7"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.453130 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:28.465189 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:28.465873 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:28.466038 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:28.473541 9617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38817
I20260927 00:53:28.473562 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38817 every 8 connection(s)
I20260927 00:53:28.474574 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:28.476614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9617
I20260927 00:53:28.476748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:53:28.477131 392 external_mini_cluster.cc:1428] Setting key df0f0a635ae00aca0592b21c2f0cd390
I20260927 00:53:28.479089 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:28.486475 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f: Bootstrap starting.
I20260927 00:53:28.487543 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:28.478976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.0.98.62:38817)
0927 00:53:28.479912 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:28.479931 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:28.479972 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:53:28.480705 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:28.480725 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:28.481191 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:28.481516 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:28.481946 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.482968 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.483851 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.484339 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.484485 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.485579 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:28.485584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:28.485598 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:28.486040 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:28.486103 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:28.486124 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:28.486213 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:28.486353 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 00:53:28.486652 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":218,"threads_started":1}
I20260927 00:53:28.489725 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:28.490744 9684 log.cc:824] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:28.493248 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f: No bootstrap required, opened a new log
I20260927 00:53:28.497700 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc51a7a0344992bab0582befbe1c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 00:53:28.498003 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:28.498050 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bc51a7a0344992bab0582befbe1c2f, State: Initialized, Role: FOLLOWER
I20260927 00:53:28.498508 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc51a7a0344992bab0582befbe1c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 00:53:28.498636 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:28.498685 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:28.498765 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:28.499956 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc51a7a0344992bab0582befbe1c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 00:53:28.500353 9684 leader_election.cc:302] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37bc51a7a0344992bab0582befbe1c2f; no voters:
I20260927 00:53:28.500722 9684 leader_election.cc:288] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:53:28.500866 9689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:28.501541 9689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [term 1 LEADER]: Becoming Leader. State: Replica: 37bc51a7a0344992bab0582befbe1c2f, State: Running, Role: LEADER
I20260927 00:53:28.501987 9689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc51a7a0344992bab0582befbe1c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 00:53:28.502288 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:28.503717 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37bc51a7a0344992bab0582befbe1c2f. Latest consensus state: current_term: 1 leader_uuid: "37bc51a7a0344992bab0582befbe1c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc51a7a0344992bab0582befbe1c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } } }
I20260927 00:53:28.503871 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:28.504225 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37bc51a7a0344992bab0582befbe1c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc51a7a0344992bab0582befbe1c2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } } }
I20260927 00:53:28.504281 9698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:28.504333 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:28.507587 9698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:28.514127 9698 catalog_manager.cc:1347] Generated new cluster ID: c37ab96130134ca1b6de4ffd35998949
I20260927 00:53:28.514216 9698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:28.521289 9698 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:53:28.522285 9698 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:28.532640 9698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37bc51a7a0344992bab0582befbe1c2f: Generated new TSK 0
I20260927 00:53:28.533321 9698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:53:28.542510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--builtin_ntp_servers=127.0.98.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 00:53:28.660127 9708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:28.660399 9708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:28.660452 9708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:28.664495 9708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:28.664558 9708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:28.664633 9708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:28.670282 9708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:28.671949 9708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:28.673696 9708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:28.684904 9713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:28.684904 9716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:28.685022 9708 server_base.cc:1034] running on GCE node
W20260927 00:53:28.684958 9714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:28.685658 9708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:28.686365 9708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:28.687529 9708 hybrid_clock.cc:648] HybridClock initialized: now 1790470408687496 us; error 52 us; skew 500 ppm
I20260927 00:53:28.689580 9708 webserver.cc:460] Webserver started at http://127.0.98.1:44389/ using document root <none> and password file <none>
I20260927 00:53:28.690173 9708 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:28.690244 9708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:28.690435 9708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:28.692523 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b1bdd1122c241b5a91687c2b55e5fb4"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "475f9bbe48e4dbdbb4f3159f59010e09"
server_key_iv: "08ac93b1b36fa79c0757da48eb447d12"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.693043 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b1bdd1122c241b5a91687c2b55e5fb4"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "475f9bbe48e4dbdbb4f3159f59010e09"
server_key_iv: "08ac93b1b36fa79c0757da48eb447d12"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.696903 9708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:53:28.699738 9722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.701329 9708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:28.701455 9708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "9b1bdd1122c241b5a91687c2b55e5fb4"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "475f9bbe48e4dbdbb4f3159f59010e09"
server_key_iv: "08ac93b1b36fa79c0757da48eb447d12"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.701597 9708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:28.729452 9708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:28.730156 9708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:28.730332 9708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:28.730916 9708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:28.731933 9708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:28.731982 9708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.732026 9708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:28.732061 9708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.741319 9708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45657
I20260927 00:53:28.741391 9835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45657 every 8 connection(s)
I20260927 00:53:28.742233 9708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:28.750782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9708
I20260927 00:53:28.750896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:53:28.751235 392 external_mini_cluster.cc:1428] Setting key 6d75b19462cef1f19ed93fb5732b2423
I20260927 00:53:28.752864 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:28.744584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56451 (local address 127.0.98.62:38817)
0927 00:53:28.744708 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:28.744711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:28.746124 (+ 1413us) server_negotiation.cc:407] Connection header received
0927 00:53:28.747611 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:28.747614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:28.747686 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:28.747760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:28.748742 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.749267 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.750379 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.750648 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.750758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.752307 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:28.752311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:28.752313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:28.752325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:28.752346 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:28.752354 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:28.752417 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:28.752638 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:53:28.752725 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:53:28.753998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--builtin_ntp_servers=127.0.98.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:53:28.754299 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:28.745041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38817 (local address 127.0.98.1:56451)
0927 00:53:28.745937 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:28.745988 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:28.747389 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:28.747948 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:28.747955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:28.748042 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:28.748602 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:28.748615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.749382 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.749385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:28.750264 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:28.750274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.750901 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.750908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:28.751189 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.751955 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:53:28.751957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:28.752017 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:53:28.752124 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:28.752461 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:28.752471 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:28.752478 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:53:28.752633 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:53:28.753020 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":154,"threads_started":1}
I20260927 00:53:28.755730 9836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38817
I20260927 00:53:28.755990 9836 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:28.756588 9836 heartbeater.cc:511] Master 127.0.98.62:38817 requested a full tablet report, sending...
I20260927 00:53:28.758513 9648 ts_manager.cc:194] Registered new tserver with Master: 9b1bdd1122c241b5a91687c2b55e5fb4 (127.0.98.1:45657)
I20260927 00:53:28.760426 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56451
W20260927 00:53:28.884286 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:28.884595 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:28.884635 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:28.888787 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:28.888865 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:28.888924 9839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:28.894019 9839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:28.895424 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:28.897100 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:53:28.908633 9839 server_base.cc:1034] running on GCE node
W20260927 00:53:28.908682 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:28.908682 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:28.908715 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:28.909493 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:28.910203 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:28.911371 9839 hybrid_clock.cc:648] HybridClock initialized: now 1790470408911351 us; error 42 us; skew 500 ppm
I20260927 00:53:28.913480 9839 webserver.cc:460] Webserver started at http://127.0.98.2:43095/ using document root <none> and password file <none>
I20260927 00:53:28.914088 9839 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:28.914155 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:28.914362 9839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:28.916543 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "e878dca3881540b7bb6e6b86e473ecc8"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "23c12c0ba6ba356da7bf17d9aead5b6e"
server_key_iv: "e03629e005e001c5c59eb2436715fd3f"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.917127 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e878dca3881540b7bb6e6b86e473ecc8"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "23c12c0ba6ba356da7bf17d9aead5b6e"
server_key_iv: "e03629e005e001c5c59eb2436715fd3f"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.921350 9839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:53:28.924306 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.925932 9839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:28.926054 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "e878dca3881540b7bb6e6b86e473ecc8"
format_stamp: "Formatted at 2026-09-27 00:53:28 on dist-test-slave-9fh2"
server_key: "23c12c0ba6ba356da7bf17d9aead5b6e"
server_key_iv: "e03629e005e001c5c59eb2436715fd3f"
server_key_version: "encryptionkey@0"
I20260927 00:53:28.926191 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:28.937438 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:28.938108 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:28.938287 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:28.938860 9839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:28.939880 9839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:28.939931 9839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.939986 9839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:28.940016 9839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:28.949385 9839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46457
I20260927 00:53:28.949441 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46457 every 8 connection(s)
I20260927 00:53:28.950438 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:28.952193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9839
I20260927 00:53:28.952325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:53:28.952661 392 external_mini_cluster.cc:1428] Setting key 09eb06218c901f478d953df384877144
I20260927 00:53:28.954962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--builtin_ntp_servers=127.0.98.20:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:53:28.961787 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:28.952813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40361 (local address 127.0.98.62:38817)
0927 00:53:28.953024 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:28.953029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:28.954267 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 00:53:28.955754 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:28.955757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:28.955830 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:28.955902 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:28.957042 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.957712 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.959414 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.959669 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.959780 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.961181 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:28.961184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:28.961187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:28.961211 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:28.961241 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:28.961251 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:28.961321 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:28.961550 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:53:28.961634 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:53:28.962978 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:28.953346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38817 (local address 127.0.98.2:40361)
0927 00:53:28.954109 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:28.954162 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:28.955541 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:28.956083 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:28.956090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:28.956185 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:28.956829 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:28.956872 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.957878 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.957881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:28.959239 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:28.959253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:28.959863 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:28.959867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:28.960092 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:28.960812 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:53:28.960814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:28.960877 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:53:28.960976 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:28.961349 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:28.961358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:28.961364 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:53:28.961518 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:53:28.961859 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":109,"threads_started":1}
I20260927 00:53:28.964859 9967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38817
I20260927 00:53:28.965195 9967 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:28.965919 9967 heartbeater.cc:511] Master 127.0.98.62:38817 requested a full tablet report, sending...
I20260927 00:53:28.967281 9648 ts_manager.cc:194] Registered new tserver with Master: e878dca3881540b7bb6e6b86e473ecc8 (127.0.98.2:46457)
I20260927 00:53:28.968154 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40361
W20260927 00:53:29.075889 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:29.076174 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:29.076213 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:29.080299 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:29.080370 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:29.080426 9970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:29.085250 9970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:29.086374 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:29.087811 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:53:29.098143 9970 server_base.cc:1034] running on GCE node
W20260927 00:53:29.098071 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.098100 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.098070 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:29.098974 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:29.099931 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:29.101142 9970 hybrid_clock.cc:648] HybridClock initialized: now 1790470409101096 us; error 65 us; skew 500 ppm
I20260927 00:53:29.103806 9970 webserver.cc:460] Webserver started at http://127.0.98.3:46413/ using document root <none> and password file <none>
I20260927 00:53:29.104583 9970 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:29.104660 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:29.104894 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:29.107581 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b1dcecab5d1433aa68720791fda6104"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "15fd48faa2169291068239fc093c6a2b"
server_key_iv: "0e6d2caebf363a2772699161c5c85d04"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.108232 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b1dcecab5d1433aa68720791fda6104"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "15fd48faa2169291068239fc093c6a2b"
server_key_iv: "0e6d2caebf363a2772699161c5c85d04"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.112901 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:53:29.116541 9984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.118197 9970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:53:29.118332 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "4b1dcecab5d1433aa68720791fda6104"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "15fd48faa2169291068239fc093c6a2b"
server_key_iv: "0e6d2caebf363a2772699161c5c85d04"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.118485 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:29.140129 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:29.141021 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:29.141216 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:29.141919 9970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:29.143060 9970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:29.143110 9970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.143181 9970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:29.143214 9970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.153641 9970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45867
I20260927 00:53:29.153676 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45867 every 8 connection(s)
I20260927 00:53:29.154752 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:29.163596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 9970
I20260927 00:53:29.163712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:53:29.163982 392 external_mini_cluster.cc:1428] Setting key 3fd762d0883cb8bb2ca813d623164001
I20260927 00:53:29.166323 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.157083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36365 (local address 127.0.98.62:38817)
0927 00:53:29.157239 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:29.157243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:29.158843 (+ 1600us) server_negotiation.cc:407] Connection header received
0927 00:53:29.160356 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:29.160359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:29.160427 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:29.160536 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:29.161549 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:29.162159 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:29.163397 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:29.163654 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:29.163775 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:29.165165 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:29.165169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:29.165173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:29.165188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:29.165213 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:29.165221 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:29.165283 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:29.166098 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 00:53:29.166172 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:53:29.166596 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.157664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38817 (local address 127.0.98.3:36365)
0927 00:53:29.158656 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:29.158708 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:29.160097 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:29.160701 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:29.160708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:29.160798 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:29.161380 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:29.161396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:29.162301 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:29.162303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:29.163251 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:29.163261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:29.163900 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:29.163906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:29.164141 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:29.164873 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:53:29.164876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:29.164935 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:53:29.165017 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:29.165314 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:29.165324 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:29.165330 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:53:29.165454 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 00:53:29.165709 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":214,"threads_started":1}
I20260927 00:53:29.167898 10098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38817
I20260927 00:53:29.168155 10098 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:29.168756 10098 heartbeater.cc:511] Master 127.0.98.62:38817 requested a full tablet report, sending...
I20260927 00:53:29.169865 9648 ts_manager.cc:194] Registered new tserver with Master: 4b1dcecab5d1433aa68720791fda6104 (127.0.98.3:45867)
I20260927 00:53:29.170540 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36365
I20260927 00:53:29.178561 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:53:29.181901 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:29.180901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.0.98.62:38817)
0927 00:53:29.181093 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:29.181098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:29.181258 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:53:29.181396 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:29.181399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:29.181446 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:29.181517 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:29.181777 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46826: BlockingRecv error: recv got EOF from 127.0.0.1:46826 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:53:29.182209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9708
I20260927 00:53:29.187480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9839
I20260927 00:53:29.192809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9970
I20260927 00:53:29.198093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 9617
2026-09-27T00:53:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (951 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T00:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:29Z Disabled control of system clock
I20260927 00:53:29.230000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42951
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:53:29.360836 10109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:29.361214 10109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:29.361259 10109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:29.365679 10109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:29.365772 10109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:29.365787 10109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:29.365803 10109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:29.365818 10109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:29.371589 10109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:29.373167 10109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:29.375097 10109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:29.385910 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.385902 10114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.385910 10115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:29.386099 10109 server_base.cc:1034] running on GCE node
I20260927 00:53:29.387001 10109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:29.388293 10109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:29.389477 10109 hybrid_clock.cc:648] HybridClock initialized: now 1790470409389459 us; error 40 us; skew 500 ppm
I20260927 00:53:29.391705 10109 webserver.cc:460] Webserver started at http://127.0.98.62:33965/ using document root <none> and password file <none>
I20260927 00:53:29.392338 10109 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:29.392403 10109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:29.392628 10109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:29.394821 10109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "52fbee48b96f43e2bcb325fce6da915e"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "8ad0fae7218122f5f97104da644da334"
server_key_iv: "21217b92ed9210a0fd4af75fed8e395b"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.395421 10109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "52fbee48b96f43e2bcb325fce6da915e"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "8ad0fae7218122f5f97104da644da334"
server_key_iv: "21217b92ed9210a0fd4af75fed8e395b"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.399601 10109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:53:29.402546 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.404122 10109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:29.404250 10109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "52fbee48b96f43e2bcb325fce6da915e"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "8ad0fae7218122f5f97104da644da334"
server_key_iv: "21217b92ed9210a0fd4af75fed8e395b"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.404374 10109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:29.409125 10109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:29.409799 10109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:29.409960 10109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:29.417363 10109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42951
I20260927 00:53:29.417413 10175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42951 every 8 connection(s)
I20260927 00:53:29.418483 10109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:29.422868 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:29.428047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10109
I20260927 00:53:29.428192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:53:29.428519 392 external_mini_cluster.cc:1428] Setting key a0fad0cd0bab08dfd35b2ef04e67891e
I20260927 00:53:29.430740 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e: Bootstrap starting.
I20260927 00:53:29.433588 10176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:29.434517 10176 log.cc:824] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:29.435436 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.430435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.62:42951)
0927 00:53:29.431458 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:29.431478 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:29.431536 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:53:29.432371 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:29.432391 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:29.432814 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:29.433160 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:29.434031 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:29.434036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:29.434049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:29.434059 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:29.434132 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:29.434149 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:29.434259 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:29.434355 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 00:53:29.434614 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":174,"threads_started":1}
I20260927 00:53:29.436898 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e: No bootstrap required, opened a new log
I20260927 00:53:29.440972 10176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fbee48b96f43e2bcb325fce6da915e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260927 00:53:29.441259 10176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:29.441295 10176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fbee48b96f43e2bcb325fce6da915e, State: Initialized, Role: FOLLOWER
I20260927 00:53:29.441804 10176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fbee48b96f43e2bcb325fce6da915e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260927 00:53:29.441955 10176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:29.442025 10176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:29.442133 10176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:29.443336 10176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fbee48b96f43e2bcb325fce6da915e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260927 00:53:29.443768 10176 leader_election.cc:302] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52fbee48b96f43e2bcb325fce6da915e; no voters:
I20260927 00:53:29.444164 10176 leader_election.cc:288] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:53:29.444334 10181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:29.444614 10181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [term 1 LEADER]: Becoming Leader. State: Replica: 52fbee48b96f43e2bcb325fce6da915e, State: Running, Role: LEADER
I20260927 00:53:29.445006 10181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fbee48b96f43e2bcb325fce6da915e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } }
I20260927 00:53:29.445637 10176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:29.446732 10182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52fbee48b96f43e2bcb325fce6da915e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fbee48b96f43e2bcb325fce6da915e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } } }
I20260927 00:53:29.446885 10182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:29.447330 10183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52fbee48b96f43e2bcb325fce6da915e. Latest consensus state: current_term: 1 leader_uuid: "52fbee48b96f43e2bcb325fce6da915e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fbee48b96f43e2bcb325fce6da915e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42951 } } }
I20260927 00:53:29.447480 10183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:29.447851 10190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:29.451046 10190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:29.457535 10190 catalog_manager.cc:1347] Generated new cluster ID: 41074317a8d74aeda7c2170368c253ed
I20260927 00:53:29.457633 10190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:29.475837 10190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:53:29.476176 10190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:29.481087 10190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52fbee48b96f43e2bcb325fce6da915e: Generated new TSK 0
I20260927 00:53:29.481794 10190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:53:29.489748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--builtin_ntp_servers=127.0.98.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:53:29.614650 10200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:29.614938 10200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:29.614979 10200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:29.619077 10200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:29.619175 10200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:29.619248 10200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:29.624053 10200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:29.625123 10200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:29.626506 10200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:29.637115 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.637121 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.637169 10205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:29.637446 10200 server_base.cc:1034] running on GCE node
I20260927 00:53:29.638236 10200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:29.639092 10200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:29.640368 10200 hybrid_clock.cc:648] HybridClock initialized: now 1790470409640293 us; error 50 us; skew 500 ppm
I20260927 00:53:29.643406 10200 webserver.cc:460] Webserver started at http://127.0.98.1:38421/ using document root <none> and password file <none>
I20260927 00:53:29.644253 10200 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:29.644342 10200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:29.644595 10200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:29.647523 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ddb77fb4bac4cfca52b7ee81bc372ba"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "15eb2973517c95161a90c2024d026e97"
server_key_iv: "bd91f75cc2b054a5a7c0396df565ad35"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.648195 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ddb77fb4bac4cfca52b7ee81bc372ba"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "15eb2973517c95161a90c2024d026e97"
server_key_iv: "bd91f75cc2b054a5a7c0396df565ad35"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.652778 10200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:53:29.656023 10214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.657650 10200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:29.657791 10200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "2ddb77fb4bac4cfca52b7ee81bc372ba"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "15eb2973517c95161a90c2024d026e97"
server_key_iv: "bd91f75cc2b054a5a7c0396df565ad35"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.657927 10200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:29.663479 10200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:29.664198 10200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:29.664374 10200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:29.664997 10200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:29.666051 10200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:29.666111 10200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.666167 10200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:29.666194 10200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.676585 10200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36025
I20260927 00:53:29.676644 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36025 every 8 connection(s)
I20260927 00:53:29.677680 10200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:29.684530 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.679720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33107 (local address 127.0.98.62:42951)
0927 00:53:29.679920 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:29.679925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:29.681169 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 00:53:29.682659 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:29.682662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:29.682706 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:29.682797 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:29.684117 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:29.684121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:29.684124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:29.684137 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:29.684162 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:29.684172 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:29.684257 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:29.684371 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 00:53:29.684408 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:53:29.685635 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.680085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42951 (local address 127.0.98.1:33107)
0927 00:53:29.680971 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:29.681025 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:29.682418 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:29.683026 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:29.683034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:29.683167 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:29.683801 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:53:29.683802 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:29.683850 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:53:29.683936 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:29.684249 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:29.684254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:29.684258 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:53:29.684413 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:53:29.684661 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":137,"threads_started":1}
I20260927 00:53:29.686971 10328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42951
I20260927 00:53:29.687269 10328 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:29.687750 10328 heartbeater.cc:511] Master 127.0.98.62:42951 requested a full tablet report, sending...
I20260927 00:53:29.687902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10200
I20260927 00:53:29.688054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:53:29.688377 392 external_mini_cluster.cc:1428] Setting key 3fc103597b56bf3c30bae828672844bd
I20260927 00:53:29.689509 10140 ts_manager.cc:194] Registered new tserver with Master: 2ddb77fb4bac4cfca52b7ee81bc372ba (127.0.98.1:36025)
I20260927 00:53:29.690654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--builtin_ntp_servers=127.0.98.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:53:29.798638 10331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:29.798914 10331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:29.798954 10331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:29.802933 10331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:29.803004 10331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:29.803059 10331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:29.807933 10331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:29.809038 10331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:29.810377 10331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:29.820952 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.820952 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:29.820966 10339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:29.821035 10331 server_base.cc:1034] running on GCE node
I20260927 00:53:29.821884 10331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:29.822763 10331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:29.824018 10331 hybrid_clock.cc:648] HybridClock initialized: now 1790470409823967 us; error 74 us; skew 500 ppm
I20260927 00:53:29.826437 10331 webserver.cc:460] Webserver started at http://127.0.98.2:41279/ using document root <none> and password file <none>
I20260927 00:53:29.827139 10331 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:29.827251 10331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:29.827509 10331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:29.830083 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce558140ffd64d4f90c224b2741c9f94"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "9cf8ea09d7b5d3a46bed2c7257c685df"
server_key_iv: "c37637d2f46568c932af22a684788178"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.830729 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce558140ffd64d4f90c224b2741c9f94"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "9cf8ea09d7b5d3a46bed2c7257c685df"
server_key_iv: "c37637d2f46568c932af22a684788178"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.835736 10331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:53:29.839625 10345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.841536 10331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:53:29.841717 10331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "ce558140ffd64d4f90c224b2741c9f94"
format_stamp: "Formatted at 2026-09-27 00:53:29 on dist-test-slave-9fh2"
server_key: "9cf8ea09d7b5d3a46bed2c7257c685df"
server_key_iv: "c37637d2f46568c932af22a684788178"
server_key_version: "encryptionkey@0"
I20260927 00:53:29.841868 10331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:29.847555 10331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:29.848343 10331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:29.848523 10331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:29.849171 10331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:29.850267 10331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:29.850317 10331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.850376 10331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:29.850405 10331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:29.861171 10331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37557
I20260927 00:53:29.861214 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37557 every 8 connection(s)
I20260927 00:53:29.862402 10331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:29.867687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10331
I20260927 00:53:29.867808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:53:29.868083 392 external_mini_cluster.cc:1428] Setting key b6d2c023fd9ff98e41c706587decaff5
I20260927 00:53:29.869764 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.864630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55965 (local address 127.0.98.62:42951)
0927 00:53:29.864834 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:29.864839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:29.866145 (+ 1306us) server_negotiation.cc:407] Connection header received
0927 00:53:29.867736 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:29.867740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:29.867797 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:29.867882 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:29.869245 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:29.869249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:29.869253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:29.869266 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:29.869288 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:29.869297 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:29.869377 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:29.869545 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:53:29.869598 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:53:29.870285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--builtin_ntp_servers=127.0.98.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:53:29.870899 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:29.864986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42951 (local address 127.0.98.2:55965)
0927 00:53:29.865899 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:29.865957 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:29.867471 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:29.868175 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:29.868186 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:29.868312 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:29.868937 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:53:29.868939 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:29.868989 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:53:29.869074 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:29.869376 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:29.869385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:29.869392 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:53:29.869605 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:53:29.869907 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":134,"threads_started":1}
I20260927 00:53:29.872316 10459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42951
I20260927 00:53:29.872663 10459 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:29.873186 10459 heartbeater.cc:511] Master 127.0.98.62:42951 requested a full tablet report, sending...
I20260927 00:53:29.874506 10140 ts_manager.cc:194] Registered new tserver with Master: ce558140ffd64d4f90c224b2741c9f94 (127.0.98.2:37557)
W20260927 00:53:29.980423 10462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:29.980738 10462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:29.980794 10462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:29.985149 10462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:29.985231 10462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:29.985292 10462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:29.990233 10462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:29.991356 10462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:29.992692 10462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:30.003470 10470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:30.003470 10468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:30.003506 10467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:30.003921 10462 server_base.cc:1034] running on GCE node
I20260927 00:53:30.004472 10462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:30.005319 10462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:30.006580 10462 hybrid_clock.cc:648] HybridClock initialized: now 1790470410006524 us; error 70 us; skew 500 ppm
I20260927 00:53:30.009485 10462 webserver.cc:460] Webserver started at http://127.0.98.3:38581/ using document root <none> and password file <none>
I20260927 00:53:30.010289 10462 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:30.010392 10462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:30.010661 10462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:30.014007 10462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "5490e8427b4c4a4fb131d2a3fe3c5dbe"
format_stamp: "Formatted at 2026-09-27 00:53:30 on dist-test-slave-9fh2"
server_key: "90bf87a2b3db08f9e0c67bbdb0e93f07"
server_key_iv: "2d6e2890ff822a6add77bd22d46c76f2"
server_key_version: "encryptionkey@0"
I20260927 00:53:30.014817 10462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5490e8427b4c4a4fb131d2a3fe3c5dbe"
format_stamp: "Formatted at 2026-09-27 00:53:30 on dist-test-slave-9fh2"
server_key: "90bf87a2b3db08f9e0c67bbdb0e93f07"
server_key_iv: "2d6e2890ff822a6add77bd22d46c76f2"
server_key_version: "encryptionkey@0"
I20260927 00:53:30.020299 10462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 00:53:30.024300 10476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:30.026152 10462 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:53:30.026356 10462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "5490e8427b4c4a4fb131d2a3fe3c5dbe"
format_stamp: "Formatted at 2026-09-27 00:53:30 on dist-test-slave-9fh2"
server_key: "90bf87a2b3db08f9e0c67bbdb0e93f07"
server_key_iv: "2d6e2890ff822a6add77bd22d46c76f2"
server_key_version: "encryptionkey@0"
I20260927 00:53:30.026508 10462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:30.032191 10462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:30.032975 10462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:30.033174 10462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:30.033838 10462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:30.034925 10462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:30.034976 10462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:30.035020 10462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:30.035034 10462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:30.045598 10462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39543
I20260927 00:53:30.045646 10589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39543 every 8 connection(s)
I20260927 00:53:30.047008 10462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:30.047539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10462
I20260927 00:53:30.047665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:53:30.047955 392 external_mini_cluster.cc:1428] Setting key ba95ad8899f122d3caec51979ac3152d
I20260927 00:53:30.054708 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:30.049236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35067 (local address 127.0.98.62:42951)
0927 00:53:30.049414 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:30.049420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:30.050981 (+ 1561us) server_negotiation.cc:407] Connection header received
0927 00:53:30.052647 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:30.052650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:30.052700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:30.052801 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:30.054133 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:30.054136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:30.054139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:53:30.054153 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:30.054178 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:53:30.054196 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:30.054296 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:30.054510 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 00:53:30.054556 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:53:30.055989 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:30.049728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42951 (local address 127.0.98.3:35067)
0927 00:53:30.050796 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:30.050866 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:30.052363 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:30.053041 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:30.053049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:30.053170 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:30.053794 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:53:30.053796 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:30.053864 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:53:30.053949 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:30.054326 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:30.054336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:30.054342 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:53:30.054534 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 00:53:30.054818 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":166,"threads_started":1}
I20260927 00:53:30.057507 10590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42951
I20260927 00:53:30.057763 10590 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:30.058223 10590 heartbeater.cc:511] Master 127.0.98.62:42951 requested a full tablet report, sending...
I20260927 00:53:30.059482 10140 ts_manager.cc:194] Registered new tserver with Master: 5490e8427b4c4a4fb131d2a3fe3c5dbe (127.0.98.3:39543)
I20260927 00:53:30.062707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:53:30.066116 10179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:53:30.065084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.0.98.62:42951)
0927 00:53:30.065276 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:30.065281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:30.065418 (+ 137us) server_negotiation.cc:407] Connection header received
0927 00:53:30.065569 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:30.065572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:30.065617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:30.065685 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:30.065953 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40620: BlockingRecv error: recv got EOF from 127.0.0.1:40620 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:53:30.066565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10200
I20260927 00:53:30.071990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10331
I20260927 00:53:30.077137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10462
I20260927 00:53:30.082875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10109
2026-09-27T00:53:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (883 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:53:30 dist-test-slave-9fh2 krb5kdc[10600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:53:30 dist-test-slave-9fh2 krb5kdc[10600](info): set up 2 sockets
Sep 27 00:53:30 dist-test-slave-9fh2 krb5kdc[10600](info): commencing operation
krb5kdc: starting...
W20260927 00:53:31.048276 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:53:31.181900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:31.291399 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:31.291688 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:31.291729 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:31.296214 10613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:53:31.296290 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:31.296310 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:31.296326 10613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:53:31.296397 10613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:53:31.301882 10613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:31.302991 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:31.304041 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:53:31.313239 10613 server_base.cc:1034] running on GCE node
W20260927 00:53:31.313174 10621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:31.313182 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:31.313164 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:31.313998 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:31.315083 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:31.316303 10613 hybrid_clock.cc:648] HybridClock initialized: now 1790470411316276 us; error 42 us; skew 500 ppm
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:31.319487 10613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:53:31.320568 10613 webserver.cc:460] Webserver started at http://127.0.98.62:33225/ using document root <none> and password file <none>
I20260927 00:53:31.321159 10613 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:31.321228 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:31.321424 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:31.323753 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "63ac9a1195af40c2974c990e48fc4e90"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "e7c9e3af482b6307f44f1e4b313b1586"
server_key_iv: "c3ab726a6aec4158ed1217c1f624c2e2"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.324344 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "63ac9a1195af40c2974c990e48fc4e90"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "e7c9e3af482b6307f44f1e4b313b1586"
server_key_iv: "c3ab726a6aec4158ed1217c1f624c2e2"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.328497 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:53:31.331451 10628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.332962 10613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:53:31.333103 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "63ac9a1195af40c2974c990e48fc4e90"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "e7c9e3af482b6307f44f1e4b313b1586"
server_key_iv: "c3ab726a6aec4158ed1217c1f624c2e2"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.333223 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:31.360550 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:31.364455 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:31.364666 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:31.372133 10613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33799
I20260927 00:53:31.372468 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33799 every 8 connection(s)
I20260927 00:53:31.373198 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:53:31.376848 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:31.378773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10613
I20260927 00:53:31.378886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:53:31.379230 392 external_mini_cluster.cc:1428] Setting key cde3c9856201492dde6534611b113fac
I20260927 00:53:31.384496 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90: Bootstrap starting.
I20260927 00:53:31.387475 10681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:31.388393 10681 log.cc:824] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:53:31.391232 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90: No bootstrap required, opened a new log
I20260927 00:53:31.394702 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:31.380899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.0.98.62:33799)
0927 00:53:31.381356 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:31.381377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:31.381593 (+ 216us) server_negotiation.cc:407] Connection header received
0927 00:53:31.382527 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:31.382545 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:31.382899 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:31.383253 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:31.384010 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.385037 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.385861 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.386351 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.386499 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:31.389559 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:31.389564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:31.389579 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:31.389607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:31.391536 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:31.392134 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:31.392140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:31.392144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:31.392297 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:31.392625 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:31.392627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:31.392629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:31.393065 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:31.393253 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:31.393558 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:53:31.393847 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260927 00:53:31.395815 10681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac9a1195af40c2974c990e48fc4e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260927 00:53:31.396112 10681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:31.396171 10681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ac9a1195af40c2974c990e48fc4e90, State: Initialized, Role: FOLLOWER
I20260927 00:53:31.396669 10681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac9a1195af40c2974c990e48fc4e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260927 00:53:31.396811 10681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:53:31.396878 10681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:53:31.396984 10681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:31.398120 10681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac9a1195af40c2974c990e48fc4e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260927 00:53:31.398495 10681 leader_election.cc:302] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ac9a1195af40c2974c990e48fc4e90; no voters:
I20260927 00:53:31.398836 10681 leader_election.cc:288] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:53:31.398976 10686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:31.399271 10686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [term 1 LEADER]: Becoming Leader. State: Replica: 63ac9a1195af40c2974c990e48fc4e90, State: Running, Role: LEADER
I20260927 00:53:31.399616 10686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac9a1195af40c2974c990e48fc4e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } }
I20260927 00:53:31.400482 10681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:53:31.401528 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ac9a1195af40c2974c990e48fc4e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac9a1195af40c2974c990e48fc4e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } } }
I20260927 00:53:31.401666 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:31.402004 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ac9a1195af40c2974c990e48fc4e90. Latest consensus state: current_term: 1 leader_uuid: "63ac9a1195af40c2974c990e48fc4e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac9a1195af40c2974c990e48fc4e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33799 } } }
I20260927 00:53:31.402097 10695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:53:31.402113 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:53:31.405069 10695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:53:31.410802 10695 catalog_manager.cc:1347] Generated new cluster ID: 82d619565e034824a1f418320aff3dce
I20260927 00:53:31.410894 10695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:53:31.422529 10695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:53:31.423579 10695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:53:31.431329 10695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90: Generated new TSK 0
I20260927 00:53:31.431962 10695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:53:31.466423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:31.578698 10707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:31.578972 10707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:31.579015 10707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:31.583251 10707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:31.583323 10707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:31.583384 10707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:53:31.588707 10707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:31.589726 10707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:31.590692 10707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:31.600648 10715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:31.600623 10712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:31.600623 10713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:31.601094 10707 server_base.cc:1034] running on GCE node
I20260927 00:53:31.601673 10707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:31.602550 10707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:31.603725 10707 hybrid_clock.cc:648] HybridClock initialized: now 1790470411603700 us; error 50 us; skew 500 ppm
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:31.607079 10707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:53:31.608238 10707 webserver.cc:460] Webserver started at http://127.0.98.1:43133/ using document root <none> and password file <none>
I20260927 00:53:31.608912 10707 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:31.608982 10707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:31.609201 10707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:31.611759 10707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "38d2cea3a18ad7b76b4b3f70a00a8102"
server_key_iv: "f0cdc5f5436ed058858c99665bb1772f"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.612381 10707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "38d2cea3a18ad7b76b4b3f70a00a8102"
server_key_iv: "f0cdc5f5436ed058858c99665bb1772f"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.617163 10707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:53:31.620457 10722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.622134 10707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:53:31.622314 10707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "38d2cea3a18ad7b76b4b3f70a00a8102"
server_key_iv: "f0cdc5f5436ed058858c99665bb1772f"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.622468 10707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:31.652707 10707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:31.656533 10707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:31.656725 10707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:31.657493 10707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:31.658631 10707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:31.658687 10707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.658735 10707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:31.658758 10707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.670222 10707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33885
I20260927 00:53:31.670274 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33885 every 8 connection(s)
I20260927 00:53:31.671450 10707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:53:31.673568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10707
I20260927 00:53:31.673748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:53:31.674110 392 external_mini_cluster.cc:1428] Setting key 12f8e4898ba0fd9d4161155a8a20ab28
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:53:31.688599 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:31.673771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60147 (local address 127.0.98.62:33799)
0927 00:53:31.673912 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:31.673916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:31.675252 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 00:53:31.677122 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:31.677125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:31.677169 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:31.677251 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:31.679120 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.679731 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.681065 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.681439 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.681559 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:31.684708 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:31.684712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:31.684715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:31.684748 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:31.686122 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:31.686729 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:31.686732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:31.686733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:31.686813 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:31.687137 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:31.687140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:31.687141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:31.687425 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:31.687531 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:31.688323 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 00:53:31.688465 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:53:31.690011 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:31.674377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33799 (local address 127.0.98.1:60147)
0927 00:53:31.675022 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:31.675084 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:31.676771 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:31.677550 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:31.677560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:31.678282 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:31.678915 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:31.678933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.679864 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.679867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:31.680911 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:31.680923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.681616 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.681618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:31.681789 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:31.682446 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:31.682466 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:31.684436 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:31.686298 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:31.686304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:31.686318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:31.686631 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:31.686891 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:31.686893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:31.686895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:31.687033 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:31.687542 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:31.687552 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:31.687976 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 00:53:31.688239 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 00:53:31.688748 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":166,"threads_started":1}
I20260927 00:53:31.691512 10836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33799
I20260927 00:53:31.691758 10836 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:31.692296 10836 heartbeater.cc:511] Master 127.0.98.62:33799 requested a full tablet report, sending...
I20260927 00:53:31.693864 10645 ts_manager.cc:194] Registered new tserver with Master: 553f3ac95bfb4fa4bd19bf4d54c93cc9 (127.0.98.1:33885)
I20260927 00:53:31.694991 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:53:31.706977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:31.837342 10841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:31.837694 10841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:31.837744 10841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:31.842305 10841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:31.842401 10841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:31.842470 10841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:53:31.849305 10841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:31.850706 10841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:31.852013 10841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:31.863279 10849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:31.863279 10846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:31.863354 10847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:31.863586 10841 server_base.cc:1034] running on GCE node
I20260927 00:53:31.864126 10841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:31.864892 10841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:31.866142 10841 hybrid_clock.cc:648] HybridClock initialized: now 1790470411866102 us; error 55 us; skew 500 ppm
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:31.869273 10841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:53:31.870229 10841 webserver.cc:460] Webserver started at http://127.0.98.2:36361/ using document root <none> and password file <none>
I20260927 00:53:31.870827 10841 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:31.870894 10841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:31.871104 10841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:31.873283 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "d1f98803992a4216a6809e2a6b0da4a3"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "029e5bbfd003e62572e37ed814d744f0"
server_key_iv: "41d2c85bb763bac5f4ac1be2b7ab9823"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.873832 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f98803992a4216a6809e2a6b0da4a3"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "029e5bbfd003e62572e37ed814d744f0"
server_key_iv: "41d2c85bb763bac5f4ac1be2b7ab9823"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.878072 10841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:53:31.880848 10856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.882359 10841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:53:31.882473 10841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "d1f98803992a4216a6809e2a6b0da4a3"
format_stamp: "Formatted at 2026-09-27 00:53:31 on dist-test-slave-9fh2"
server_key: "029e5bbfd003e62572e37ed814d744f0"
server_key_iv: "41d2c85bb763bac5f4ac1be2b7ab9823"
server_key_version: "encryptionkey@0"
I20260927 00:53:31.882566 10841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:31.893828 10841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:31.896960 10841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:31.897219 10841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:31.897825 10841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:31.898656 10841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:31.898705 10841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.898756 10841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:31.898785 10841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:31.908746 10841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33037
I20260927 00:53:31.908777 10969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33037 every 8 connection(s)
I20260927 00:53:31.909667 10841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:53:31.915392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10841
I20260927 00:53:31.915535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:53:31.915889 392 external_mini_cluster.cc:1428] Setting key 28b47195fa29cc0f58c954f23efd6eda
Sep 27 00:53:31 dist-test-slave-9fh2 krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:53:31.925525 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:31.912008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48125 (local address 127.0.98.62:33799)
0927 00:53:31.912176 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:31.912181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:31.913156 (+ 975us) server_negotiation.cc:407] Connection header received
0927 00:53:31.914580 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:31.914583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:31.914635 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:31.914726 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:31.916564 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.917146 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.918510 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.918736 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.918824 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:31.921691 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:31.921694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:31.921696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:31.921722 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:31.923037 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:31.923695 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:31.923700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:31.923701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:31.923829 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:31.924241 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:31.924243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:31.924244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:31.924461 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:31.924545 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:31.925220 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 00:53:31.925364 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:53:31.926687 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:31.912399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33799 (local address 127.0.98.2:48125)
0927 00:53:31.912928 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:31.912985 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:31.914355 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:31.914899 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:31.914907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:31.915524 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:31.916378 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:31.916399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.917316 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.917322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:31.918380 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:31.918394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:31.918882 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:31.918886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:31.919071 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:31.919692 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:31.919709 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:31.921521 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:31.923174 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:31.923181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:31.923199 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:31.923535 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:31.923938 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:31.923941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:31.923943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:31.924144 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:31.924565 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:31.924573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:31.924941 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 00:53:31.925161 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 00:53:31.925534 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:53:31.928107 10970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33799
I20260927 00:53:31.928567 10970 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:31.929253 10970 heartbeater.cc:511] Master 127.0.98.62:33799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:53:31.930738 10645 ts_manager.cc:194] Registered new tserver with Master: d1f98803992a4216a6809e2a6b0da4a3 (127.0.98.2:33037)
I20260927 00:53:31.931708 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:53:31.950143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:53:32.070169 10975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:53:32.070470 10975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:53:32.070528 10975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:53:32.074697 10975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:53:32.074769 10975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:53:32.074855 10975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:53:32.080288 10975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:53:32.081377 10975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:53:32.082403 10975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:53:32.092814 10983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:32.092795 10980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:53:32.092795 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:53:32.093295 10975 server_base.cc:1034] running on GCE node
I20260927 00:53:32.093848 10975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:53:32.094671 10975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:53:32.095855 10975 hybrid_clock.cc:648] HybridClock initialized: now 1790470412095827 us; error 47 us; skew 500 ppm
Sep 27 00:53:32 dist-test-slave-9fh2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:53:32.104466 10975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:53:32.105682 10975 webserver.cc:460] Webserver started at http://127.0.98.3:37609/ using document root <none> and password file <none>
I20260927 00:53:32.106384 10975 fs_manager.cc:359] Metadata directory not provided
I20260927 00:53:32.106454 10975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:53:32.106693 10975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:53:32.109273 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e0eb8e12f704fcd80230918fcb7d375"
format_stamp: "Formatted at 2026-09-27 00:53:32 on dist-test-slave-9fh2"
server_key: "a54b8a79e30bab1fadb5cf8a87184b8b"
server_key_iv: "1247f0691a6e62a736904068e62e03fd"
server_key_version: "encryptionkey@0"
I20260927 00:53:32.109899 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e0eb8e12f704fcd80230918fcb7d375"
format_stamp: "Formatted at 2026-09-27 00:53:32 on dist-test-slave-9fh2"
server_key: "a54b8a79e30bab1fadb5cf8a87184b8b"
server_key_iv: "1247f0691a6e62a736904068e62e03fd"
server_key_version: "encryptionkey@0"
I20260927 00:53:32.114511 10975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 00:53:32.118155 10990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:32.119938 10975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:53:32.120079 10975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "2e0eb8e12f704fcd80230918fcb7d375"
format_stamp: "Formatted at 2026-09-27 00:53:32 on dist-test-slave-9fh2"
server_key: "a54b8a79e30bab1fadb5cf8a87184b8b"
server_key_iv: "1247f0691a6e62a736904068e62e03fd"
server_key_version: "encryptionkey@0"
I20260927 00:53:32.120216 10975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:53:32.137754 10975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:53:32.141770 10975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:53:32.141956 10975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:53:32.142618 10975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:53:32.143667 10975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:53:32.143720 10975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:32.143765 10975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:53:32.143779 10975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:53:32.154834 10975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42223
I20260927 00:53:32.154853 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42223 every 8 connection(s)
I20260927 00:53:32.156005 10975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:53:32.158671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 10975
I20260927 00:53:32.158797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:53:32.159090 392 external_mini_cluster.cc:1428] Setting key 8f61a053c9218135879fe5a0ad3261a1
Sep 27 00:53:32 dist-test-slave-9fh2 krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:53:32.171855 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.158308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58621 (local address 127.0.98.62:33799)
0927 00:53:32.158526 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.158532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.159611 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 00:53:32.161125 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.161129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.161197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.161272 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:32.162685 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.163321 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.164563 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.164861 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.164959 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.167857 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:32.167860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:32.167862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:32.167890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:32.169430 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:32.170037 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:32.170041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:32.170042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:32.170120 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:32.170472 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:32.170476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:32.170478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:32.170702 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:32.170801 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.171503 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.171657 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 00:53:32.173117 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.158737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33799 (local address 127.0.98.3:58621)
0927 00:53:32.159420 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.159481 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:32.160880 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:32.161498 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:32.161506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:32.161939 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:53:32.162527 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.162543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.163455 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.163459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.164436 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.164448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.165026 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.165031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.165225 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.165891 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:53:32.165910 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:53:32.167657 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:53:32.169595 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:32.169601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:32.169619 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:32.169933 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:32.170210 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:53:32.170213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:53:32.170216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:53:32.170379 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:53:32.170815 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:53:32.170823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:53:32.171219 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 00:53:32.171444 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 00:53:32.171879 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":121,"threads_started":1}
I20260927 00:53:32.174451 11104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33799
I20260927 00:53:32.174787 11104 heartbeater.cc:464] Registering TS with master...
I20260927 00:53:32.175493 11104 heartbeater.cc:511] Master 127.0.98.62:33799 requested a full tablet report, sending...
I20260927 00:53:32.176932 10645 ts_manager.cc:194] Registered new tserver with Master: 2e0eb8e12f704fcd80230918fcb7d375 (127.0.98.3:42223)
I20260927 00:53:32.177628 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58621
I20260927 00:53:32.185421 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:53:32.196177 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.187805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.62:33799)
0927 00:53:32.187955 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.187959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.188177 (+ 218us) server_negotiation.cc:407] Connection header received
0927 00:53:32.188319 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.188322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.188370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.188465 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:53:32.189331 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.189891 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.190883 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.191108 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.191259 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.192795 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:53:32.192798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:53:32.192800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:53:32.192824 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:53:32.194320 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:32.194845 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:32.194848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:32.194855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:32.194948 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:53:32.195234 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:53:32.195237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:53:32.195239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:53:32.195559 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:53:32.195676 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.195934 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.196043 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:53:32.199683 10645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:53:32.202924 10645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:53:32.219099 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.212610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33037 (local address 127.0.0.1:43964)
0927 00:53:32.213921 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.213934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:32.214001 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:32.215104 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:32.215107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:32.215124 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:32.215281 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.215288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.216442 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.216445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.217543 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.217552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.218500 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.218503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.218617 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.218675 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 00:53:32.218795 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:53:32.218940 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"thread_start_us":136,"threads_started":1}
I20260927 00:53:32.219192 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.213146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.0.98.2:33037)
0927 00:53:32.214732 (+ 1586us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.214745 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.214768 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:53:32.214839 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.214843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.214978 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.215104 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:32.215394 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.216294 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.217673 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.218368 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.218481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.218661 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.218853 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.218973 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1473,"thread_start_us":128,"threads_started":1}
I20260927 00:53:32.219660 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.212465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42223 (local address 127.0.0.1:45418)
0927 00:53:32.213506 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.213562 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:32.213709 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:32.214438 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:32.214442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:32.214469 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:32.214670 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.214678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.216220 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.216223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.218021 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.218033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.219227 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.219230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.219334 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.219363 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:53:32.219424 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:53:32.219510 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":362,"spinlock_wait_cycles":6272,"thread_start_us":122,"threads_started":1}
I20260927 00:53:32.219856 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.212416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33885 (local address 127.0.0.1:46916)
0927 00:53:32.213506 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.213562 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:32.213717 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:32.214426 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:32.214433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:32.214482 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:32.214679 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.214687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.216213 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.216217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.218021 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.218033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.219206 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.219209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.219499 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.219526 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:53:32.219590 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:53:32.219680 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":240,"spinlock_wait_cycles":40576,"thread_start_us":127,"threads_started":1}
I20260927 00:53:32.220248 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.213434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.0.98.1:33885)
0927 00:53:32.213893 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.213903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.213931 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:53:32.214027 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.214033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.214251 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.214400 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:32.214850 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.216070 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.218172 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.218883 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.218973 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.219121 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.220041 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.220137 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":82,"threads_started":1}
I20260927 00:53:32.220357 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.212575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.0.98.3:42223)
0927 00:53:32.213987 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.213998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.214022 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:53:32.214107 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.214112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.214291 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.214436 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:32.214802 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.216070 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.218203 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.219018 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.219124 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.220061 (+ 937us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.220112 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.220210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309,"thread_start_us":172,"threads_started":1}
I20260927 00:53:32.222966 10770 tablet_service.cc:1467] Processing CreateTablet for tablet bd19137a70f64ff5a7cc5969882eb3fb (DEFAULT_TABLE table=test-table [id=f9105c27893e4b758e2db5b0b0e90623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:32.223079 10904 tablet_service.cc:1467] Processing CreateTablet for tablet bd19137a70f64ff5a7cc5969882eb3fb (DEFAULT_TABLE table=test-table [id=f9105c27893e4b758e2db5b0b0e90623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:32.224027 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd19137a70f64ff5a7cc5969882eb3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:32.224345 10904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd19137a70f64ff5a7cc5969882eb3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:32.225709 11038 tablet_service.cc:1467] Processing CreateTablet for tablet bd19137a70f64ff5a7cc5969882eb3fb (DEFAULT_TABLE table=test-table [id=f9105c27893e4b758e2db5b0b0e90623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:53:32.226940 11038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd19137a70f64ff5a7cc5969882eb3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:53:32.230587 11121 tablet_bootstrap.cc:492] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9: Bootstrap starting.
I20260927 00:53:32.232750 11122 tablet_bootstrap.cc:492] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: Bootstrap starting.
I20260927 00:53:32.233105 11121 tablet_bootstrap.cc:654] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:32.234016 11121 log.cc:824] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:32.235404 11122 tablet_bootstrap.cc:654] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:32.236236 11122 log.cc:824] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:32.236702 11121 tablet_bootstrap.cc:492] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9: No bootstrap required, opened a new log
I20260927 00:53:32.236696 11124 tablet_bootstrap.cc:492] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Bootstrap starting.
I20260927 00:53:32.236912 11121 ts_tablet_manager.cc:1397] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:53:32.238786 11122 tablet_bootstrap.cc:492] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: No bootstrap required, opened a new log
I20260927 00:53:32.239068 11122 ts_tablet_manager.cc:1397] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:53:32.240005 11124 tablet_bootstrap.cc:654] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Neither blocks nor log segments found. Creating new log.
I20260927 00:53:32.240846 11121 raft_consensus.cc:348] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.241011 11124 log.cc:824] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Log is configured to *not* fsync() on all Append() calls
I20260927 00:53:32.241081 11121 raft_consensus.cc:374] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:32.241123 11121 raft_consensus.cc:729] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553f3ac95bfb4fa4bd19bf4d54c93cc9, State: Initialized, Role: FOLLOWER
I20260927 00:53:32.241513 11121 consensus_queue.cc:260] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.242501 11121 ts_tablet_manager.cc:1428] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:53:32.242811 11128 raft_consensus.cc:482] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:53:32.242868 10836 heartbeater.cc:503] Master 127.0.98.62:33799 was elected leader, sending a full tablet report...
I20260927 00:53:32.242935 11128 raft_consensus.cc:504] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.243539 11122 raft_consensus.cc:348] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.243834 11124 tablet_bootstrap.cc:492] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: No bootstrap required, opened a new log
I20260927 00:53:32.243877 11122 raft_consensus.cc:374] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:32.243934 11122 raft_consensus.cc:729] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f98803992a4216a6809e2a6b0da4a3, State: Initialized, Role: FOLLOWER
I20260927 00:53:32.244048 11124 ts_tablet_manager.cc:1397] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:53:32.244451 11122 consensus_queue.cc:260] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.245584 11122 ts_tablet_manager.cc:1428] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:53:32.246017 10970 heartbeater.cc:503] Master 127.0.98.62:33799 was elected leader, sending a full tablet report...
I20260927 00:53:32.247175 11128 leader_election.cc:288] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e0eb8e12f704fcd80230918fcb7d375 (127.0.98.3:42223), d1f98803992a4216a6809e2a6b0da4a3 (127.0.98.2:33037)
I20260927 00:53:32.248217 11124 raft_consensus.cc:348] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.248605 11124 raft_consensus.cc:374] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:53:32.248669 11124 raft_consensus.cc:729] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e0eb8e12f704fcd80230918fcb7d375, State: Initialized, Role: FOLLOWER
I20260927 00:53:32.249514 11124 consensus_queue.cc:260] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.250818 11124 ts_tablet_manager.cc:1428] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:53:32.251181 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.246406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58981 (local address 127.0.98.3:42223)
0927 00:53:32.246541 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.246546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.246564 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:53:32.246702 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.246706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.246781 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.246833 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:32.247357 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.248152 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.249729 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.250324 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.250408 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.250502 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.250923 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.251012 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:53:32.251276 11104 heartbeater.cc:503] Master 127.0.98.62:33799 was elected leader, sending a full tablet report...
I20260927 00:53:32.251662 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.246008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42223 (local address 127.0.98.1:58981)
0927 00:53:32.246370 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.246394 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:32.246547 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:32.247020 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:32.247023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:32.247041 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:32.247224 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.247232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.248304 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.248307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.249587 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.249599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.250605 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.250609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.250751 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.250784 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:53:32.251386 (+ 602us) client_negotiation.cc:240] Negotiation successful
0927 00:53:32.251492 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260927 00:53:32.252345 11058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd19137a70f64ff5a7cc5969882eb3fb" candidate_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e0eb8e12f704fcd80230918fcb7d375" is_pre_election: true
I20260927 00:53:32.252825 11058 raft_consensus.cc:2454] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553f3ac95bfb4fa4bd19bf4d54c93cc9 in term 0.
I20260927 00:53:32.253356 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.246916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33037 (local address 127.0.98.1:33475)
0927 00:53:32.247411 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.247424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:53:32.247521 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:53:32.249559 (+ 2038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:53:32.249562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:53:32.249581 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:53:32.249737 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.249745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.250700 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.250712 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.251872 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:53:32.251882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.252846 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.252849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:53:32.252953 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.252980 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:53:32.253078 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:53:32.253175 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":31,"thread_start_us":82,"threads_started":1}
I20260927 00:53:32.253393 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.249108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33475 (local address 127.0.98.2:33037)
0927 00:53:32.249254 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.249259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.249273 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:53:32.249321 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.249324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.249366 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.249440 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:53:32.249863 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.250586 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.251999 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.252589 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.252668 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.252750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.253142 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.253259 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:53:32.253396 10725 leader_election.cc:302] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e0eb8e12f704fcd80230918fcb7d375, 553f3ac95bfb4fa4bd19bf4d54c93cc9; no voters:
I20260927 00:53:32.253726 11128 raft_consensus.cc:2781] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:53:32.253830 11128 raft_consensus.cc:482] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:53:32.253873 11128 raft_consensus.cc:3037] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:32.254140 10924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd19137a70f64ff5a7cc5969882eb3fb" candidate_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f98803992a4216a6809e2a6b0da4a3" is_pre_election: true
I20260927 00:53:32.254464 10924 raft_consensus.cc:2454] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553f3ac95bfb4fa4bd19bf4d54c93cc9 in term 0.
I20260927 00:53:32.255133 11128 raft_consensus.cc:504] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.255573 11128 leader_election.cc:288] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [CANDIDATE]: Term 1 election: Requested vote from peers 2e0eb8e12f704fcd80230918fcb7d375 (127.0.98.3:42223), d1f98803992a4216a6809e2a6b0da4a3 (127.0.98.2:33037)
I20260927 00:53:32.255965 11058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd19137a70f64ff5a7cc5969882eb3fb" candidate_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e0eb8e12f704fcd80230918fcb7d375"
I20260927 00:53:32.255944 10924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd19137a70f64ff5a7cc5969882eb3fb" candidate_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f98803992a4216a6809e2a6b0da4a3"
I20260927 00:53:32.256155 10924 raft_consensus.cc:3037] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:32.256131 11058 raft_consensus.cc:3037] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:53:32.257454 11058 raft_consensus.cc:2454] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553f3ac95bfb4fa4bd19bf4d54c93cc9 in term 1.
I20260927 00:53:32.257498 10924 raft_consensus.cc:2454] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553f3ac95bfb4fa4bd19bf4d54c93cc9 in term 1.
I20260927 00:53:32.257834 10726 leader_election.cc:302] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553f3ac95bfb4fa4bd19bf4d54c93cc9, d1f98803992a4216a6809e2a6b0da4a3; no voters:
I20260927 00:53:32.258042 11128 raft_consensus.cc:2781] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:53:32.258173 11128 raft_consensus.cc:686] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 553f3ac95bfb4fa4bd19bf4d54c93cc9, State: Running, Role: LEADER
I20260927 00:53:32.258589 11128 consensus_queue.cc:237] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } }
I20260927 00:53:32.262104 10644 catalog_manager.cc:5495] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 553f3ac95bfb4fa4bd19bf4d54c93cc9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553f3ac95bfb4fa4bd19bf4d54c93cc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 } health_report { overall_health: UNKNOWN } } }
I20260927 00:53:32.280965 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.277008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.0.98.1:33885)
0927 00:53:32.277214 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 00:53:32.277219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:53:32.277240 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:53:32.277331 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:53:32.277333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:53:32.277399 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:53:32.277504 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:53:32.277790 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.278293 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.279355 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:53:32.279577 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:53:32.279674 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:53:32.280069 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:53:32.280543 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:53:32.280642 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:53:32.280765 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 00:53:32.280823 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:53:32.288259 11058 raft_consensus.cc:1261] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 1 FOLLOWER]: Refusing update from remote peer 553f3ac95bfb4fa4bd19bf4d54c93cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:32.288300 10924 raft_consensus.cc:1261] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 1 FOLLOWER]: Refusing update from remote peer 553f3ac95bfb4fa4bd19bf4d54c93cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:53:32.289230 11127 consensus_queue.cc:1035] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f98803992a4216a6809e2a6b0da4a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:32.289443 11128 consensus_queue.cc:1035] T bd19137a70f64ff5a7cc5969882eb3fb P 553f3ac95bfb4fa4bd19bf4d54c93cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e0eb8e12f704fcd80230918fcb7d375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:53:32.296952 11135 mvcc.cc:205] Tried to move back new op lower bound from 7333766808723107840 to 7333766808614436864. Current Snapshot: MvccSnapshot[applied={T|T < 7333766808723107840}]
I20260927 00:53:32.297699 11137 mvcc.cc:205] Tried to move back new op lower bound from 7333766808723107840 to 7333766808614436864. Current Snapshot: MvccSnapshot[applied={T|T < 7333766808723107840}]
I20260927 00:53:32.299713 11136 mvcc.cc:205] Tried to move back new op lower bound from 7333766808723107840 to 7333766808614436864. Current Snapshot: MvccSnapshot[applied={T|T < 7333766808723107840}]
I20260927 00:53:32.307855 10644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:32.308037 10644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:53:32.310789 10644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63ac9a1195af40c2974c990e48fc4e90: Sending DeleteTablet for 3 replicas of tablet bd19137a70f64ff5a7cc5969882eb3fb
I20260927 00:53:32.311743 10904 tablet_service.cc:1514] Processing DeleteTablet for tablet bd19137a70f64ff5a7cc5969882eb3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43964
I20260927 00:53:32.311769 11038 tablet_service.cc:1514] Processing DeleteTablet for tablet bd19137a70f64ff5a7cc5969882eb3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45418
I20260927 00:53:32.311872 10770 tablet_service.cc:1514] Processing DeleteTablet for tablet bd19137a70f64ff5a7cc5969882eb3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:53:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46916
I20260927 00:53:32.312400 11147 tablet_replica.cc:330] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: stopping tablet replica
I20260927 00:53:32.312750 11147 raft_consensus.cc:2229] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:32.312815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10707
I20260927 00:53:32.313725 11145 tablet_replica.cc:330] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: stopping tablet replica
I20260927 00:53:32.314118 11145 raft_consensus.cc:2229] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:53:32.314486 11145 raft_consensus.cc:2258] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:32.313079 11147 raft_consensus.cc:2258] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:53:32.316813 11147 ts_tablet_manager.cc:1905] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:53:32.319001 11147 ts_tablet_manager.cc:1918] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:53:32.319093 11147 log.cc:1196] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/bd19137a70f64ff5a7cc5969882eb3fb
I20260927 00:53:32.319443 11147 ts_tablet_manager.cc:1939] T bd19137a70f64ff5a7cc5969882eb3fb P 2e0eb8e12f704fcd80230918fcb7d375: Deleting consensus metadata
I20260927 00:53:32.320226 11145 ts_tablet_manager.cc:1905] T bd19137a70f64ff5a7cc5969882eb3fb P d1f98803992a4216a6809e2a6b0da4a3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:53:32.320550 10630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33885 (error 108)
I20260927 00:53:32.320989 10631 catalog_manager.cc:4841] TS 2e0eb8e12f704fcd80230918fcb7d375 (127.0.98.3:42223): tablet bd19137a70f64ff5a7cc5969882eb3fb (table test-table [id=f9105c27893e4b758e2db5b0b0e90623]) successfully deleted
I20260927 00:53:32.321475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10841
I20260927 00:53:32.322250 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.321715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33885 (local address 127.0.0.1:46930)
0927 00:53:32.322078 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.322155 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273}
W20260927 00:53:32.322470 10630 catalog_manager.cc:4584] TS 553f3ac95bfb4fa4bd19bf4d54c93cc9 (127.0.98.1:33885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd19137a70f64ff5a7cc5969882eb3fb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33885: connect: Connection refused (error 111)
I20260927 00:53:32.327314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10975
I20260927 00:53:32.327826 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:53:32.327497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33037 (local address 127.0.0.1:43980)
0927 00:53:32.327694 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 00:53:32.327739 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 00:53:32.332621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 10613
2026-09-27T00:53:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2250 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:53:32 dist-test-slave-9fh2 krb5kdc[11151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:53:32 dist-test-slave-9fh2 krb5kdc[11151](info): set up 2 sockets
Sep 27 00:53:32 dist-test-slave-9fh2 krb5kdc[11151](info): commencing operation
krb5kdc: starting...
W20260927 00:53:33.300433 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:53:33 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:53:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:53:33.490294 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskj1hhZs/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:53:36.179 UTC [11181] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:53:36.179 UTC [11181] LOG: listening on IPv4 address "127.0.98.20", port 47009
2026-09-27 00:53:36.184 UTC [11181] LOG: listening on Unix socket "/tmp/.s.PGSQL.47009"
2026-09-27 00:53:36.206 UTC [11184] LOG: database system was shut down at 2026-09-27 00:53:34 UTC
2026-09-27 00:53:36.211 UTC [11181] LOG: database system is ready to accept connections
I20260927 00:53:37.088060 392 mini_postgres.cc:96] Postgres bound to 47009
2026-09-27 00:53:37.095 UTC [11192] FATAL: database "slave" does not exist
127.0.98.20:47009 - accepting connections
I20260927 00:53:37.095961 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:53:37.111670 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:53:37.645735 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:53:37.645845 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-admin
2026-09-27 00:53:37,934 [I] DB FLAVOR :POSTGRES
2026-09-27 00:53:37,934 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:53:37,934 [I] Checking connection..
2026-09-27 00:53:37,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:53:38,202 [I] Checking connection passed.
2026-09-27 00:53:38,202 [I] --------- Verifying version history table ---------
2026-09-27 00:53:38,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:53:38,462 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:53:38,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:53:38,728 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:53:38,729 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:53:38,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:53:38.996 UTC [11256] WARNING: there is no transaction in progress
2026-09-27 00:53:39,003 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:53:39,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:53:39,287 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:53:39,288 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:53:39,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:53:39,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:53:39,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9fh2.c.gcp-upstream.internal','N') ;"
2026-09-27 00:53:40,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:53:40,369 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:53:40,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:53:40,656 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:53:40,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:53:40,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:53:41,186 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:53:41,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:53:42.275 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.294 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.313 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.330 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.350 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.369 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.434 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.442 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.457 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.469 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.481 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.496 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.508 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.515 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.522 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:42.529 UTC [11364] WARNING: there is no transaction in progress
2026-09-27 00:53:43,284 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:53:43,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:53:43,529 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:53:43,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:53:43,790 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:53:43,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:53:44,066 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:53:44,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:53:44,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:53:44,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9fh2.c.gcp-upstream.internal';"
2026-09-27 00:53:44,875 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:53:44,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:53:45,145 [I] DB_PATCHES have already been applied
I20260927 00:53:45.151119 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskj1hhZs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:53:45.151254 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:53:45.155344 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52201
Sep 27, 2026 12:53:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:53:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:53:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:53:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:53:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:53:46.161526 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.066s user 0.000s sys 0.012s
Sep 27, 2026 12:54:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:54:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:54:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:54:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:54:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:54:10.068378 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:54:10 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470450, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:54:10.180308 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:54:10.190097 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:54:11.146981 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:54:11.147100 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms
I20260927 00:54:11.147255 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:47009
2026-09-27 00:54:11,375 [I] DB FLAVOR :POSTGRES
2026-09-27 00:54:11,375 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:54:11,375 [I] Checking connection
2026-09-27 00:54:11,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:54:11,644 [I] connection success
2026-09-27 00:54:11,645 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:54:11,645 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:54:11,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:54:11,942 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:54:11,943 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:54:11,943 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:54:11,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:54:12,260 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:54:12.781540 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:54:12.989717 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:54:13.046065 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:54:13 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470450, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:54:13.448041 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:54:13.448220 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:54:13.448243 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:54:13.452881 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260927 00:54:13.453012 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35775
Sep 27, 2026 12:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:54:15.523401 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.343s user 0.020s sys 0.194s
I20260927 00:54:15.523582 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:54:15.523727 392 mini_ranger_kms.cc:401] 127.0.98.20:35775/kms/v1/keys
Sep 27 00:54:15 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470455, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:54:16 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470455, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:54:16 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470455, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:54:18 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:54:18.794349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38671
--ranger_config_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35775/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:54:18.928283 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:54:18.928709 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:54:18.928756 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:54:18.933194 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:54:18.933295 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:54:18.933314 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:54:18.933332 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:54:18.933348 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:54:18.939704 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35775/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38671
--ranger_config_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:54:18.941139 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:54:18.942392 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:54:18.952775 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:54:18.952888 11595 server_base.cc:1034] running on GCE node
W20260927 00:54:18.952778 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:18.952775 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:54:18.953807 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:54:18.955186 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:54:18.956367 11595 hybrid_clock.cc:648] HybridClock initialized: now 1790470458956346 us; error 41 us; skew 500 ppm
Sep 27 00:54:18 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:54:18.959520 11595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:54:18.960604 11595 webserver.cc:460] Webserver started at http://127.0.98.62:37631/ using document root <none> and password file <none>
I20260927 00:54:18.961228 11595 fs_manager.cc:359] Metadata directory not provided
I20260927 00:54:18.961287 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:54:18.961516 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:54:18 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:54:19.023702 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "450d496901934c329f3592f1f4447fe6"
format_stamp: "Formatted at 2026-09-27 00:54:19 on dist-test-slave-9fh2"
server_key: "27c3fa5345fe3a6afb9939101fbd75c8"
server_key_iv: "27d1b6d14836950eeff1661b4895ef69"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:19.024488 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "450d496901934c329f3592f1f4447fe6"
format_stamp: "Formatted at 2026-09-27 00:54:19 on dist-test-slave-9fh2"
server_key: "27c3fa5345fe3a6afb9939101fbd75c8"
server_key_iv: "27d1b6d14836950eeff1661b4895ef69"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:19.029393 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:54:19.048098 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:19.049510 11595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:54:19.049647 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "450d496901934c329f3592f1f4447fe6"
format_stamp: "Formatted at 2026-09-27 00:54:19 on dist-test-slave-9fh2"
server_key: "27c3fa5345fe3a6afb9939101fbd75c8"
server_key_iv: "27d1b6d14836950eeff1661b4895ef69"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:19.049787 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:54:19.061018 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:54:19.064021 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:54:19.064203 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:54:19.071777 11595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38671
I20260927 00:54:19.071844 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38671 every 8 connection(s)
I20260927 00:54:19.072849 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:54:19.077432 11663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:54:19.082691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 11595
I20260927 00:54:19.082825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:54:19.083106 392 external_mini_cluster.cc:1428] Setting key 0de9d0796fd41040d1b3133a35975fe2
I20260927 00:54:19.087076 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6: Bootstrap starting.
I20260927 00:54:19.091022 11663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6: Neither blocks nor log segments found. Creating new log.
I20260927 00:54:19.092224 11663 log.cc:824] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:54:19 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:54:19.095114 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6: No bootstrap required, opened a new log
I20260927 00:54:19.099057 11663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d496901934c329f3592f1f4447fe6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38671 } }
I20260927 00:54:19.099416 11663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:54:19.099471 11663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450d496901934c329f3592f1f4447fe6, State: Initialized, Role: FOLLOWER
I20260927 00:54:19.099828 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:19.084771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.0.98.62:38671)
0927 00:54:19.085227 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:19.085243 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:19.085280 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:54:19.086579 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:19.086603 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:19.086998 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:19.087458 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:54:19.088246 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:19.089461 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:19.090291 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:19.090779 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:19.090925 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:19.093392 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:54:19.093399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:54:19.093418 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:54:19.093448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:54:19.096209 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:19.096945 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:19.096952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:19.096958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:19.097091 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:19.097429 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:19.097433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:19.097434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:19.097949 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:54:19.098144 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:19.098496 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 00:54:19.098812 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":116,"threads_started":1}
I20260927 00:54:19.099994 11663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d496901934c329f3592f1f4447fe6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38671 } }
I20260927 00:54:19.100169 11663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:54:19.100276 11663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:54:19.100381 11663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:54:19.101549 11663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d496901934c329f3592f1f4447fe6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38671 } }
I20260927 00:54:19.101933 11663 leader_election.cc:302] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 450d496901934c329f3592f1f4447fe6; no voters:
I20260927 00:54:19.102291 11663 leader_election.cc:288] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:54:19.102378 11668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:54:19.102988 11668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [term 1 LEADER]: Becoming Leader. State: Replica: 450d496901934c329f3592f1f4447fe6, State: Running, Role: LEADER
I20260927 00:54:19.103385 11668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d496901934c329f3592f1f4447fe6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38671 } }
I20260927 00:54:19.103807 11663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:54:19.104575 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450d496901934c329f3592f1f4447fe6. Latest consensus state: current_term: 1 leader_uuid: "450d496901934c329f3592f1f4447fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d496901934c329f3592f1f4447fe6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38671 } } }
I20260927 00:54:19.105389 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:54:19.105273 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "450d496901934c329f3592f1f4447fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d496901934c329f3592f1f4447fe6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38671 } } }
I20260927 00:54:19.105710 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:54:19.106123 11663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:54:19.109686 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:54:19.114648 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:54:19.125254 11681 catalog_manager.cc:1347] Generated new cluster ID: b24a3aa329bc4db2beb5d33109a69287
I20260927 00:54:19.125339 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:54:19.148310 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:54:19.149699 11681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:54:19.160481 11681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6: Generated new TSK 0
I20260927 00:54:19.161476 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:54:19.910974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35775/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38671
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:54:20.073513 11708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:54:20.074050 11708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:54:20.074165 11708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:54:20.080621 11708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:54:20.080821 11708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:54:20.080960 11708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:54:20.088487 11708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35775/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:54:20.089941 11708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:54:20.091348 11708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:54:20.102973 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:20.102885 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:20.103188 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:54:20.103744 11708 server_base.cc:1034] running on GCE node
I20260927 00:54:20.104233 11708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:54:20.105101 11708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:54:20.106276 11708 hybrid_clock.cc:648] HybridClock initialized: now 1790470460106247 us; error 51 us; skew 500 ppm
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:54:20.109668 11708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:54:20.110726 11708 webserver.cc:460] Webserver started at http://127.0.98.1:36095/ using document root <none> and password file <none>
I20260927 00:54:20.111423 11708 fs_manager.cc:359] Metadata directory not provided
I20260927 00:54:20.111490 11708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:54:20.111713 11708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:54:20.140142 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "253416660ce14ae38c13385f51982810"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "2225d935b5a685fb66c1a50c1532ebdb"
server_key_iv: "c910c1723d298ab68fbbe9a39d407992"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.141235 11708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "253416660ce14ae38c13385f51982810"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "2225d935b5a685fb66c1a50c1532ebdb"
server_key_iv: "c910c1723d298ab68fbbe9a39d407992"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.154960 11708 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260927 00:54:20.186965 11727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.189265 11708 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 00:54:20.189502 11708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "253416660ce14ae38c13385f51982810"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "2225d935b5a685fb66c1a50c1532ebdb"
server_key_iv: "c910c1723d298ab68fbbe9a39d407992"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.189695 11708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:54:20.208863 11708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:54:20.212329 11708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:54:20.212544 11708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:54:20.213263 11708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:54:20.214287 11708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:54:20.214336 11708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.214395 11708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:54:20.214437 11708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.226737 11708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38457
I20260927 00:54:20.226755 11840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38457 every 8 connection(s)
I20260927 00:54:20.228150 11708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:54:20.230924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 11708
I20260927 00:54:20.231063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:54:20.231480 392 external_mini_cluster.cc:1428] Setting key 080ff31f9f8cafd14ceb8f263f18c1f1
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:54:20.250347 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.230461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55537 (local address 127.0.98.62:38671)
0927 00:54:20.236182 (+ 5721us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:20.236188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:20.236208 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:54:20.236263 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:20.236266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:20.236333 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:20.237985 (+ 1652us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:54:20.238093 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.238680 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.240489 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.240863 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.240990 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.245435 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:54:20.245440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:54:20.245443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:54:20.245495 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:54:20.247353 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.247975 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.247978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.247980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.248061 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.248427 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.248430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.248432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.248662 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:54:20.248747 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:20.249443 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 00:54:20.250129 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":68,"threads_started":1}
I20260927 00:54:20.254614 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.231037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38671 (local address 127.0.98.1:55537)
0927 00:54:20.232486 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:20.232551 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:20.233722 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:20.236777 (+ 3055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:20.236789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:20.237181 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:54:20.237833 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:20.237850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.238828 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.238833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:20.240307 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:20.240323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.241163 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.241171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:20.241495 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.243070 (+ 1575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:54:20.243100 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:54:20.245103 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:54:20.247502 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:54:20.247511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:54:20.247537 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:54:20.247855 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:54:20.248148 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:54:20.248152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:54:20.248154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:54:20.248326 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:54:20.248767 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:54:20.248776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:54:20.249172 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 00:54:20.249375 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:54:20.249868 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":169,"threads_started":1}
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:54:20.257557 11841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38671
I20260927 00:54:20.257933 11841 heartbeater.cc:464] Registering TS with master...
I20260927 00:54:20.258776 11841 heartbeater.cc:511] Master 127.0.98.62:38671 requested a full tablet report, sending...
I20260927 00:54:20.261317 11627 ts_manager.cc:194] Registered new tserver with Master: 253416660ce14ae38c13385f51982810 (127.0.98.1:38457)
I20260927 00:54:20.262773 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:54:20.271610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35775/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38671
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:54:20.420004 11847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:54:20.420377 11847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:54:20.420454 11847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:54:20.426100 11847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:54:20.426221 11847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:54:20.426313 11847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:54:20.432952 11847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35775/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:54:20.434794 11847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:54:20.436754 11847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:54:20.451083 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:20.451017 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:20.451508 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:54:20.452688 11847 server_base.cc:1034] running on GCE node
I20260927 00:54:20.453370 11847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:54:20.454312 11847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:54:20.455621 11847 hybrid_clock.cc:648] HybridClock initialized: now 1790470460455587 us; error 61 us; skew 500 ppm
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:54:20.459842 11847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:54:20.461098 11847 webserver.cc:460] Webserver started at http://127.0.98.2:32959/ using document root <none> and password file <none>
I20260927 00:54:20.461851 11847 fs_manager.cc:359] Metadata directory not provided
I20260927 00:54:20.461926 11847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:54:20.462179 11847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:54:20.481941 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "0dc83a149936479ea059bfc51fafb908"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "05d172a323b306ac94dac3f2d5ad2a80"
server_key_iv: "cd70259f77985a0a66247c9c2e95f5ff"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.483417 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0dc83a149936479ea059bfc51fafb908"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "05d172a323b306ac94dac3f2d5ad2a80"
server_key_iv: "cd70259f77985a0a66247c9c2e95f5ff"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.488175 11847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:54:20.513280 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.515065 11847 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 00:54:20.515363 11847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "0dc83a149936479ea059bfc51fafb908"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "05d172a323b306ac94dac3f2d5ad2a80"
server_key_iv: "cd70259f77985a0a66247c9c2e95f5ff"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.515540 11847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:54:20.577415 11847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:54:20.581117 11847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:54:20.581321 11847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:54:20.582010 11847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:54:20.583071 11847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:54:20.583139 11847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.583243 11847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:54:20.583281 11847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.595683 11847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36981
I20260927 00:54:20.595774 11976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36981 every 8 connection(s)
I20260927 00:54:20.596925 11847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:54:20.600662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 11847
I20260927 00:54:20.600819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:54:20.601192 392 external_mini_cluster.cc:1428] Setting key 2ffb588909992c86bef0e9d8ff8700aa
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:54:20.616144 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.601990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57935 (local address 127.0.98.62:38671)
0927 00:54:20.602319 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:20.602324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:20.602478 (+ 154us) server_negotiation.cc:407] Connection header received
0927 00:54:20.603781 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:20.603785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:20.603864 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:20.603921 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:54:20.605517 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.606404 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.608079 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.608412 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.608491 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.611684 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:54:20.611689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:54:20.611692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:54:20.611724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:54:20.613483 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.614159 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.614163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.614165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.614259 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.614684 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.614687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.614689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.614941 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:54:20.615027 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:20.615790 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 00:54:20.615956 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260927 00:54:20.617996 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.600834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38671 (local address 127.0.98.2:57935)
0927 00:54:20.602274 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:20.602357 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:20.603531 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:20.604338 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:20.604346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:20.604704 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:54:20.605351 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:20.605369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.606553 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.606556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:20.607949 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:20.607960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.608712 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.608716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:20.608925 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.609614 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:54:20.609634 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:54:20.611458 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:54:20.613634 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:54:20.613640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:54:20.613658 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:54:20.614037 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:54:20.614373 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:54:20.614375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:54:20.614378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:54:20.614570 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:54:20.615094 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:54:20.615101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:54:20.615499 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 00:54:20.616343 (+ 844us) client_negotiation.cc:240] Negotiation successful
0927 00:54:20.616717 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:54:20.619614 11977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38671
I20260927 00:54:20.619889 11977 heartbeater.cc:464] Registering TS with master...
I20260927 00:54:20.620476 11977 heartbeater.cc:511] Master 127.0.98.62:38671 requested a full tablet report, sending...
I20260927 00:54:20.621685 11627 ts_manager.cc:194] Registered new tserver with Master: 0dc83a149936479ea059bfc51fafb908 (127.0.98.2:36981)
I20260927 00:54:20.622524 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:54:20.638775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35775/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38671
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:54:20.808204 11982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:54:20.808578 11982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:54:20.808691 11982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:54:20.814491 11982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:54:20.814577 11982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:54:20.814726 11982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:54:20.822343 11982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35775/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:54:20.823853 11982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:54:20.825207 11982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:54:20.838688 11987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:20.839186 11990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:54:20.839356 11988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:54:20.839423 11982 server_base.cc:1034] running on GCE node
I20260927 00:54:20.840016 11982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:54:20.840818 11982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:54:20.842006 11982 hybrid_clock.cc:648] HybridClock initialized: now 1790470460841942 us; error 79 us; skew 500 ppm
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:54:20.845456 11982 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:54:20.846510 11982 webserver.cc:460] Webserver started at http://127.0.98.3:36835/ using document root <none> and password file <none>
I20260927 00:54:20.847234 11982 fs_manager.cc:359] Metadata directory not provided
I20260927 00:54:20.847323 11982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:54:20.847554 11982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:54:20.883745 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "7472319a358b442ba4d060c0c592a2ca"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "e73af613043aba986647920b827ad811"
server_key_iv: "bf83f57c9bc8847c0106934a86276396"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.884507 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7472319a358b442ba4d060c0c592a2ca"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "e73af613043aba986647920b827ad811"
server_key_iv: "bf83f57c9bc8847c0106934a86276396"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.889935 11982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 00:54:20.910192 11997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.912004 11982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:54:20.912173 11982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "7472319a358b442ba4d060c0c592a2ca"
format_stamp: "Formatted at 2026-09-27 00:54:20 on dist-test-slave-9fh2"
server_key: "e73af613043aba986647920b827ad811"
server_key_iv: "bf83f57c9bc8847c0106934a86276396"
server_key_version: "kuduclusterkey@0"
I20260927 00:54:20.912377 11982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:54:20.937031 11982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:54:20.940462 11982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:54:20.940678 11982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:54:20.941406 11982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:54:20.942405 11982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:54:20.942464 11982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.942559 11982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:54:20.942585 11982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:54:20.955250 11982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46047
I20260927 00:54:20.955485 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46047 every 8 connection(s)
I20260927 00:54:20.956594 11982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:54:20.960176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 11982
I20260927 00:54:20.960316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:54:20.960654 392 external_mini_cluster.cc:1428] Setting key cd10dc392e1090b24c6db821a850f23b
Sep 27 00:54:20 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:54:20.972973 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.958804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55119 (local address 127.0.98.62:38671)
0927 00:54:20.958964 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:20.958969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:20.961709 (+ 2740us) server_negotiation.cc:407] Connection header received
0927 00:54:20.963101 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:20.963105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:20.963190 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:20.963242 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:54:20.964819 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.965396 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.966599 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.966837 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.966924 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.969468 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:54:20.969471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:54:20.969473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:54:20.969499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:54:20.970795 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.971374 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.971379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.971381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.971459 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.971793 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.971796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.971797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.971990 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:54:20.972074 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:20.972649 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 00:54:20.972785 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:54:20.974709 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.959440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38671 (local address 127.0.98.3:55119)
0927 00:54:20.961543 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:20.961610 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:20.962841 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:20.963665 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:20.963674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:20.964003 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:54:20.964666 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:20.964683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.965515 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.965519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:20.966475 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:20.966485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.966984 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.966988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:20.967222 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.967875 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:54:20.967894 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:54:20.969296 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:54:20.970901 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:54:20.970907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:54:20.970926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:54:20.971262 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:54:20.971537 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:54:20.971540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:54:20.971542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:54:20.971696 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:54:20.972083 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:54:20.972090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:54:20.972396 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 00:54:20.972582 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:54:20.972949 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1815,"thread_start_us":162,"threads_started":1}
I20260927 00:54:20.976002 12111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38671
I20260927 00:54:20.976279 12111 heartbeater.cc:464] Registering TS with master...
I20260927 00:54:20.976866 12111 heartbeater.cc:511] Master 127.0.98.62:38671 requested a full tablet report, sending...
I20260927 00:54:20.977942 11627 ts_manager.cc:194] Registered new tserver with Master: 7472319a358b442ba4d060c0c592a2ca (127.0.98.3:46047)
I20260927 00:54:20.978582 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55119
I20260927 00:54:20.986981 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:54:20.999406 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:20.989469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.0.98.62:38671)
0927 00:54:20.989610 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:20.989615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:20.989834 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:54:20.989968 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:20.989971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:20.990026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:20.990092 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:54:20.990814 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.991490 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.992341 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:20.992570 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:20.992658 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:20.994235 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:54:20.994239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:54:20.994241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:54:20.994266 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:54:20.995718 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.996111 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.996115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.996117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.996193 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:54:20.996436 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:54:20.996439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:54:20.996441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:54:20.996659 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:54:20.998928 (+ 2269us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:20.999023 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:54:20.999207 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:54:21.003578 11627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:54:21.008527 11627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:54:21.030323 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.021423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46047 (local address 127.0.0.1:53546)
0927 00:54:21.022781 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.022794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.022890 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.023748 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:21.023755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:21.023777 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:54:21.023941 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.023949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.025475 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.025478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.028182 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.028193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.029600 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.029604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.029802 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.029847 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 00:54:21.029967 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:54:21.030120 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":106,"thread_start_us":98,"threads_started":1}
I20260927 00:54:21.031107 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.022063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.0.98.3:46047)
0927 00:54:21.022641 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:21.022655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:21.022987 (+ 332us) server_negotiation.cc:407] Connection header received
0927 00:54:21.023084 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:21.023090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:21.023359 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:21.023553 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:54:21.024081 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.025334 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.028348 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.029242 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.029360 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.029534 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:21.030797 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0927 00:54:21.030945 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":160,"threads_started":1}
I20260927 00:54:21.031409 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.021955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.0.98.2:36981)
0927 00:54:21.023351 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:21.023369 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:21.023676 (+ 307us) server_negotiation.cc:407] Connection header received
0927 00:54:21.023772 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:21.023777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:21.023957 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:21.024093 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:54:21.024498 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.026689 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.028349 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.029242 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.029350 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.029534 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:21.031040 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0927 00:54:21.031194 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1293,"thread_start_us":148,"threads_started":1}
I20260927 00:54:21.030933 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.021711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36981 (local address 127.0.0.1:44738)
0927 00:54:21.023502 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.023514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.023596 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.024239 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:21.024242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:21.024258 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:54:21.024342 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.024347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.026813 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.026817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.028182 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.028193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.029600 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.029604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.030631 (+ 1027us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.030656 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:54:21.030720 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:54:21.030797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1732,"mutex_wait_us":76,"thread_start_us":111,"threads_started":1}
I20260927 00:54:21.035526 12045 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d5f1079bf4281b0e77f190cf2f1d4 (DEFAULT_TABLE table=test-table [id=e209887037024992b9bfec9b3858e419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:54:21.035526 11893 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d5f1079bf4281b0e77f190cf2f1d4 (DEFAULT_TABLE table=test-table [id=e209887037024992b9bfec9b3858e419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:54:21.036955 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d5f1079bf4281b0e77f190cf2f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:54:21.037866 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d5f1079bf4281b0e77f190cf2f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:54:21.040139 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.020051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38457 (local address 127.0.0.1:39996)
0927 00:54:21.020511 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.020566 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.020706 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.030858 (+ 10152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:21.030863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:21.030910 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:54:21.031080 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.031091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.032919 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.032924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.034240 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.034252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.039610 (+ 5358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.039616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.039742 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.039784 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:54:21.039845 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:54:21.039947 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":137,"threads_started":1}
I20260927 00:54:21.041249 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.020285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.0.98.1:38457)
0927 00:54:21.027747 (+ 7462us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:21.027757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:21.027792 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:54:21.027905 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:21.027913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:21.028185 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:21.031356 (+ 3171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:54:21.031535 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.032766 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.034406 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.039383 (+ 4977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.040699 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.040899 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:21.040966 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 00:54:21.041077 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7296,"thread_start_us":117,"threads_started":1}
I20260927 00:54:21.046739 11759 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d5f1079bf4281b0e77f190cf2f1d4 (DEFAULT_TABLE table=test-table [id=e209887037024992b9bfec9b3858e419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:54:21.048278 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d5f1079bf4281b0e77f190cf2f1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:54:21.048727 12129 tablet_bootstrap.cc:492] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Bootstrap starting.
I20260927 00:54:21.051620 12129 tablet_bootstrap.cc:654] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Neither blocks nor log segments found. Creating new log.
I20260927 00:54:21.053447 12129 log.cc:824] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Log is configured to *not* fsync() on all Append() calls
I20260927 00:54:21.056373 12129 tablet_bootstrap.cc:492] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: No bootstrap required, opened a new log
I20260927 00:54:21.056627 12129 ts_tablet_manager.cc:1397] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:54:21.062119 12129 raft_consensus.cc:348] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.062537 12129 raft_consensus.cc:374] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:54:21.062602 12129 raft_consensus.cc:729] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc83a149936479ea059bfc51fafb908, State: Initialized, Role: FOLLOWER
I20260927 00:54:21.063582 12129 consensus_queue.cc:260] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.065038 12129 ts_tablet_manager.cc:1428] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:54:21.065563 11977 heartbeater.cc:503] Master 127.0.98.62:38671 was elected leader, sending a full tablet report...
I20260927 00:54:21.066972 12133 tablet_bootstrap.cc:492] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Bootstrap starting.
I20260927 00:54:21.067457 12131 tablet_bootstrap.cc:492] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: Bootstrap starting.
I20260927 00:54:21.073848 12133 tablet_bootstrap.cc:654] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Neither blocks nor log segments found. Creating new log.
I20260927 00:54:21.075636 12131 tablet_bootstrap.cc:654] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: Neither blocks nor log segments found. Creating new log.
I20260927 00:54:21.078457 12131 log.cc:824] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: Log is configured to *not* fsync() on all Append() calls
I20260927 00:54:21.079541 12133 log.cc:824] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Log is configured to *not* fsync() on all Append() calls
I20260927 00:54:21.082687 12133 tablet_bootstrap.cc:492] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: No bootstrap required, opened a new log
I20260927 00:54:21.082937 12133 ts_tablet_manager.cc:1397] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260927 00:54:21.087021 12131 tablet_bootstrap.cc:492] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: No bootstrap required, opened a new log
I20260927 00:54:21.088995 12131 ts_tablet_manager.cc:1397] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260927 00:54:21.096405 12133 raft_consensus.cc:348] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.096633 12131 raft_consensus.cc:348] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.096868 12131 raft_consensus.cc:374] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:54:21.096923 12131 raft_consensus.cc:729] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253416660ce14ae38c13385f51982810, State: Initialized, Role: FOLLOWER
I20260927 00:54:21.097031 12133 raft_consensus.cc:374] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:54:21.097118 12133 raft_consensus.cc:729] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7472319a358b442ba4d060c0c592a2ca, State: Initialized, Role: FOLLOWER
I20260927 00:54:21.097587 12131 consensus_queue.cc:260] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.097585 12133 consensus_queue.cc:260] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.098938 12133 ts_tablet_manager.cc:1428] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Time spent starting tablet: real 0.016s user 0.002s sys 0.005s
W20260927 00:54:21.099220 11978 tablet.cc:2369] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:54:21.099516 12111 heartbeater.cc:503] Master 127.0.98.62:38671 was elected leader, sending a full tablet report...
I20260927 00:54:21.098939 12131 ts_tablet_manager.cc:1428] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260927 00:54:21.100366 11841 heartbeater.cc:503] Master 127.0.98.62:38671 was elected leader, sending a full tablet report...
Sep 27 00:54:21 dist-test-slave-9fh2 krb5kdc[11151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:54:21.147921 12138 raft_consensus.cc:482] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:54:21.148207 12138 raft_consensus.cc:504] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.150774 12138 leader_election.cc:288] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253416660ce14ae38c13385f51982810 (127.0.98.1:38457), 0dc83a149936479ea059bfc51fafb908 (127.0.98.2:36981)
I20260927 00:54:21.155505 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.149880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38457 (local address 127.0.98.3:51983)
0927 00:54:21.150078 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.150096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.150214 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.151397 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:21.151400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:21.151432 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:54:21.151585 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.151594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.152607 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.152611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.153886 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.153893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.155001 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.155005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.155109 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.155135 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:54:21.155227 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:54:21.155314 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 00:54:21.155980 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.150256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51983 (local address 127.0.98.1:38457)
0927 00:54:21.150961 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:21.150966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:21.150982 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:54:21.151043 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:21.151046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:21.151102 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:21.151363 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:54:21.151725 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.152489 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.154107 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.154755 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.154853 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.154938 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:21.155741 (+ 803us) server_negotiation.cc:299] Negotiation successful
0927 00:54:21.155850 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602}
I20260927 00:54:21.157169 11778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d5f1079bf4281b0e77f190cf2f1d4" candidate_uuid: "7472319a358b442ba4d060c0c592a2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253416660ce14ae38c13385f51982810" is_pre_election: true
I20260927 00:54:21.157469 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.150795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44637 (local address 127.0.98.2:36981)
0927 00:54:21.151288 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:21.151293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:21.151822 (+ 529us) server_negotiation.cc:407] Connection header received
0927 00:54:21.151926 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:21.151930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:21.151997 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:21.152088 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:54:21.152407 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.153167 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.155626 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.156233 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.156315 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.156381 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:21.157227 (+ 846us) server_negotiation.cc:299] Negotiation successful
0927 00:54:21.157333 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260927 00:54:21.157585 11778 raft_consensus.cc:2454] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7472319a358b442ba4d060c0c592a2ca in term 0.
I20260927 00:54:21.158408 11999 leader_election.cc:302] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253416660ce14ae38c13385f51982810, 7472319a358b442ba4d060c0c592a2ca; no voters:
I20260927 00:54:21.158790 12138 raft_consensus.cc:2781] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:54:21.159009 12138 raft_consensus.cc:482] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:54:21.159257 12138 raft_consensus.cc:3037] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:54:21.160212 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.150477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36981 (local address 127.0.98.3:44637)
0927 00:54:21.151708 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.151722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.151831 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.152127 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:54:21.152130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:54:21.152151 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:54:21.152274 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.152281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.153884 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.153887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.155489 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:54:21.155499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.156865 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.156868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:54:21.157053 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.157080 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:54:21.158810 (+ 1730us) client_negotiation.cc:240] Negotiation successful
0927 00:54:21.158911 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":148,"threads_started":1}
I20260927 00:54:21.161151 12138 raft_consensus.cc:504] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.161108 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d5f1079bf4281b0e77f190cf2f1d4" candidate_uuid: "7472319a358b442ba4d060c0c592a2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc83a149936479ea059bfc51fafb908" is_pre_election: true
I20260927 00:54:21.161463 11931 raft_consensus.cc:2454] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7472319a358b442ba4d060c0c592a2ca in term 0.
I20260927 00:54:21.161630 12138 leader_election.cc:288] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [CANDIDATE]: Term 1 election: Requested vote from peers 253416660ce14ae38c13385f51982810 (127.0.98.1:38457), 0dc83a149936479ea059bfc51fafb908 (127.0.98.2:36981)
I20260927 00:54:21.162135 11778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d5f1079bf4281b0e77f190cf2f1d4" candidate_uuid: "7472319a358b442ba4d060c0c592a2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253416660ce14ae38c13385f51982810"
I20260927 00:54:21.162305 11778 raft_consensus.cc:3037] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:54:21.163445 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d5f1079bf4281b0e77f190cf2f1d4" candidate_uuid: "7472319a358b442ba4d060c0c592a2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc83a149936479ea059bfc51fafb908"
I20260927 00:54:21.163553 11931 raft_consensus.cc:3037] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:54:21.164733 11931 raft_consensus.cc:2454] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7472319a358b442ba4d060c0c592a2ca in term 1.
I20260927 00:54:21.165083 11999 leader_election.cc:302] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc83a149936479ea059bfc51fafb908, 7472319a358b442ba4d060c0c592a2ca; no voters:
I20260927 00:54:21.165402 12138 raft_consensus.cc:2781] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:54:21.165501 11778 raft_consensus.cc:2454] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7472319a358b442ba4d060c0c592a2ca in term 1.
I20260927 00:54:21.165728 12138 raft_consensus.cc:686] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 1 LEADER]: Becoming Leader. State: Replica: 7472319a358b442ba4d060c0c592a2ca, State: Running, Role: LEADER
I20260927 00:54:21.166581 12138 consensus_queue.cc:237] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } }
I20260927 00:54:21.171562 11627 catalog_manager.cc:5495] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 7472319a358b442ba4d060c0c592a2ca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7472319a358b442ba4d060c0c592a2ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7472319a358b442ba4d060c0c592a2ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 } health_report { overall_health: UNKNOWN } } }
I20260927 00:54:21.190873 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.184860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.0.98.3:46047)
0927 00:54:21.185045 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:54:21.185051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:54:21.185070 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:54:21.185200 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:54:21.185203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:54:21.185293 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:54:21.185446 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:54:21.185960 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.186763 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.188024 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:54:21.188411 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:54:21.188538 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:54:21.188975 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:54:21.189806 (+ 831us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:54:21.189951 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:54:21.190598 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 00:54:21.190685 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:54:21.201905 11931 raft_consensus.cc:1261] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 1 FOLLOWER]: Refusing update from remote peer 7472319a358b442ba4d060c0c592a2ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:54:21.201905 11778 raft_consensus.cc:1261] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810 [term 1 FOLLOWER]: Refusing update from remote peer 7472319a358b442ba4d060c0c592a2ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:54:21.202858 12138 consensus_queue.cc:1035] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "253416660ce14ae38c13385f51982810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:54:21.203178 12140 consensus_queue.cc:1035] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc83a149936479ea059bfc51fafb908" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:54:21.211623 12112 tablet.cc:2369] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:54:21.227182 12144 mvcc.cc:205] Tried to move back new op lower bound from 7333767009067692032 to 7333767008943181824. Current Snapshot: MvccSnapshot[applied={T|T < 7333767009067692032}]
W20260927 00:54:21.230229 11842 tablet.cc:2369] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 253416660ce14ae38c13385f51982810: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:54:21.249320 12147 mvcc.cc:205] Tried to move back new op lower bound from 7333767009067692032 to 7333767008943181824. Current Snapshot: MvccSnapshot[applied={T|T < 7333767009067692032}]
I20260927 00:54:21.251037 12145 mvcc.cc:205] Tried to move back new op lower bound from 7333767009067692032 to 7333767008943181824. Current Snapshot: MvccSnapshot[applied={T|T < 7333767009067692032 or (T in {7333767009067692032})}]
I20260927 00:54:21.278291 11626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:54:21.278605 11626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:54:21.283169 11626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 450d496901934c329f3592f1f4447fe6: Sending DeleteTablet for 3 replicas of tablet fb9d5f1079bf4281b0e77f190cf2f1d4
I20260927 00:54:21.285125 11759 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d5f1079bf4281b0e77f190cf2f1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:54:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39996
I20260927 00:54:21.286880 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d5f1079bf4281b0e77f190cf2f1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:54:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44738
I20260927 00:54:21.288161 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d5f1079bf4281b0e77f190cf2f1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:54:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53546
I20260927 00:54:21.290258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 11708
I20260927 00:54:21.303807 12155 tablet_replica.cc:330] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: stopping tablet replica
I20260927 00:54:21.304203 12154 tablet_replica.cc:330] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: stopping tablet replica
I20260927 00:54:21.304950 12154 raft_consensus.cc:2229] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:54:21.307844 12155 raft_consensus.cc:2229] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:54:21.308640 12154 raft_consensus.cc:2258] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:54:21.309147 12155 raft_consensus.cc:2258] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:54:21.311561 12155 ts_tablet_manager.cc:1905] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:54:21.313063 12154 ts_tablet_manager.cc:1905] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:54:21.320936 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38457 (error 108)
I20260927 00:54:21.319922 12154 ts_tablet_manager.cc:1918] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 7472319a358b442ba4d060c0c592a2ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:54:21.322407 12155 ts_tablet_manager.cc:1918] T fb9d5f1079bf4281b0e77f190cf2f1d4 P 0dc83a149936479ea059bfc51fafb908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:54:21.322541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 11847
I20260927 00:54:21.325366 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:54:21.324876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38457 (local address 127.0.0.1:40008)
0927 00:54:21.325152 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.325243 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260927 00:54:21.332305 11612 catalog_manager.cc:4584] TS 253416660ce14ae38c13385f51982810 (127.0.98.1:38457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb9d5f1079bf4281b0e77f190cf2f1d4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38457: connect: Connection refused (error 111)
W20260927 00:54:21.335882 12125 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:54:21.335113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36981 (local address 127.0.0.1:44744)
0927 00:54:21.335366 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.335384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.335507 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.335720 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 00:54:21.336807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 11982
W20260927 00:54:21.349431 12125 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:54:21.348610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46047 (local address 127.0.0.1:53572)
0927 00:54:21.348929 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:54:21.348948 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:54:21.349081 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:54:21.349280 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:46047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":187}
I20260927 00:54:21.350504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 11595
I20260927 00:54:21.359236 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:54:21.694859 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:54:21.695034 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:54:22.031106 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:54:22.031 UTC [11181] LOG: received smart shutdown request
2026-09-27 00:54:22.035 UTC [11181] LOG: background worker "logical replication launcher" (PID 11190) exited with exit code 1
2026-09-27 00:54:22.036 UTC [11185] LOG: shutting down
2026-09-27 00:54:22.223 UTC [11181] LOG: database system is shut down
2026-09-27T00:54:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49981 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:54:22 dist-test-slave-9fh2 krb5kdc[12165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:54:22 dist-test-slave-9fh2 krb5kdc[12165](info): set up 2 sockets
Sep 27 00:54:22 dist-test-slave-9fh2 krb5kdc[12165](info): commencing operation
krb5kdc: starting...
W20260927 00:54:23.287602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:54:23 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:54:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:54:23.474434 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskj1hhZs/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:54:26.123 UTC [12195] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:54:26.123 UTC [12195] LOG: listening on IPv4 address "127.0.98.20", port 43297
2026-09-27 00:54:26.127 UTC [12195] LOG: listening on Unix socket "/tmp/.s.PGSQL.43297"
2026-09-27 00:54:26.148 UTC [12198] LOG: database system was shut down at 2026-09-27 00:54:24 UTC
2026-09-27 00:54:26.151 UTC [12195] LOG: database system is ready to accept connections
I20260927 00:54:27.019408 392 mini_postgres.cc:96] Postgres bound to 43297
2026-09-27 00:54:27.026 UTC [12206] FATAL: database "slave" does not exist
127.0.98.20:43297 - accepting connections
I20260927 00:54:27.027006 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:54:27.043267 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:54:27.594250 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:54:27.594336 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-admin
2026-09-27 00:54:27,880 [I] DB FLAVOR :POSTGRES
2026-09-27 00:54:27,880 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:54:27,880 [I] Checking connection..
2026-09-27 00:54:27,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:54:28,143 [I] Checking connection passed.
2026-09-27 00:54:28,143 [I] --------- Verifying version history table ---------
2026-09-27 00:54:28,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:54:28,418 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:54:28,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:54:28,681 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:54:28,682 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:54:28,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:54:28.942 UTC [12270] WARNING: there is no transaction in progress
2026-09-27 00:54:28,949 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:54:28,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:54:29,217 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:54:29,217 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:54:29,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:54:29,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:54:29,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9fh2.c.gcp-upstream.internal','N') ;"
2026-09-27 00:54:30,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:54:30,296 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:54:30,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:54:30,549 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:54:30,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:54:30,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:54:31,120 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:54:31,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:54:32.255 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.275 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.294 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.314 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.337 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.362 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.424 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.433 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.449 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.461 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.472 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.490 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.502 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.510 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.518 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:32.526 UTC [12378] WARNING: there is no transaction in progress
2026-09-27 00:54:33,317 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:54:33,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:54:33,584 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:54:33,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:54:33,848 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:54:33,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:54:34,113 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:54:34,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:54:34,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:54:34,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9fh2.c.gcp-upstream.internal';"
2026-09-27 00:54:34,958 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:54:34,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:54:35,247 [I] DB_PATCHES have already been applied
I20260927 00:54:35.253638 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskj1hhZs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:54:35.253741 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:54:35.257831 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48907
Sep 27, 2026 12:54:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:54:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:54:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:54:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:54:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:54:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:54:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:54:37.345644 392 mini_ranger.cc:160] Time spent starting Ranger: real 10.319s user 0.005s sys 0.009s
Sep 27, 2026 12:54:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:54:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:54:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:54:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:54:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:55:00.132208 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:55:00 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470500, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:55:00.242148 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:55:00.254276 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:55:01.209908 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:55:01.210045 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms
I20260927 00:55:01.210064 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43297
2026-09-27 00:55:01,408 [I] DB FLAVOR :POSTGRES
2026-09-27 00:55:01,408 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:55:01,409 [I] Checking connection
2026-09-27 00:55:01,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:55:01,664 [I] connection success
2026-09-27 00:55:01,664 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:55:01,664 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:55:01,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:55:01,956 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:55:01,957 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:55:01,957 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:55:01,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43297/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:55:02,277 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:55:02.833204 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:55:03.029975 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:55:03.084939 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:55:03 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470500, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:03.535214 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:55:03.535370 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskj1hhZs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskj1hhZs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:55:03.535394 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:55:03.540107 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12560
I20260927 00:55:03.540189 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48331
Sep 27, 2026 12:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:55:05.566689 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.325s user 0.020s sys 0.164s
I20260927 00:55:05.566807 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:55:05.566874 392 mini_ranger_kms.cc:401] 127.0.98.20:48331/kms/v1/keys
Sep 27 00:55:05 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470505, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:55:06 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470505, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:55:06 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470505, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:55:08 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:55:08.966988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055
--ranger_config_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:09.137957 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:09.138335 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:09.138396 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:09.142877 12610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:09.142982 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:09.143009 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:09.143030 12610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:09.143051 12610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:09.149439 12610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48331/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055
--ranger_config_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:09.150924 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:09.152266 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:09.162405 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:09.162406 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:09.162406 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:09.163271 12610 server_base.cc:1034] running on GCE node
I20260927 00:55:09.164063 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:09.165565 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:09.166751 12610 hybrid_clock.cc:648] HybridClock initialized: now 1790470509166719 us; error 72 us; skew 500 ppm
Sep 27 00:55:09 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:09.170380 12610 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:09.171758 12610 webserver.cc:460] Webserver started at http://127.0.98.62:38305/ using document root <none> and password file <none>
I20260927 00:55:09.172466 12610 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:09.172536 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:09.172777 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:55:09 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:09.230386 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "f4eff57a13b847159044b7175c2fc576"
format_stamp: "Formatted at 2026-09-27 00:55:09 on dist-test-slave-9fh2"
server_key: "88fbc178511d3acebd840a3e07aad2e8"
server_key_iv: "9e2d91886013150af7785629c731b891"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:09.231294 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4eff57a13b847159044b7175c2fc576"
format_stamp: "Formatted at 2026-09-27 00:55:09 on dist-test-slave-9fh2"
server_key: "88fbc178511d3acebd840a3e07aad2e8"
server_key_iv: "9e2d91886013150af7785629c731b891"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:09.236780 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:55:09.254967 12625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:09.256784 12610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:55:09.257002 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "f4eff57a13b847159044b7175c2fc576"
format_stamp: "Formatted at 2026-09-27 00:55:09 on dist-test-slave-9fh2"
server_key: "88fbc178511d3acebd840a3e07aad2e8"
server_key_iv: "9e2d91886013150af7785629c731b891"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:09.257186 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:09.278355 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:09.281930 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:09.282155 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:09.291042 12610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37055
I20260927 00:55:09.291082 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37055 every 8 connection(s)
I20260927 00:55:09.292309 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:09.295511 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:09.298139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 12610
I20260927 00:55:09.298291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:55:09.298687 392 external_mini_cluster.cc:1428] Setting key a2d1eb527b3710e497ae20142d80f8c2
I20260927 00:55:09.301613 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Bootstrap starting.
I20260927 00:55:09.304227 12678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:09.305002 12678 log.cc:824] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:09.307335 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: No bootstrap required, opened a new log
Sep 27 00:55:09 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:09.311556 12678 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:09.311877 12678 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:09.311936 12678 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eff57a13b847159044b7175c2fc576, State: Initialized, Role: FOLLOWER
I20260927 00:55:09.312448 12678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:09.312619 12678 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:09.312698 12678 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:09.312805 12678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:09.313941 12678 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:09.314311 12678 leader_election.cc:302] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4eff57a13b847159044b7175c2fc576; no voters:
I20260927 00:55:09.314684 12678 leader_election.cc:288] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:55:09.314842 12683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:09.315127 12683 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 LEADER]: Becoming Leader. State: Replica: f4eff57a13b847159044b7175c2fc576, State: Running, Role: LEADER
I20260927 00:55:09.315717 12683 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:09.315995 12678 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:09.316134 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:09.300630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.0.98.62:37055)
0927 00:55:09.301127 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:09.301146 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:09.301188 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:55:09.302036 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:09.302065 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:09.302475 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:09.302816 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:09.303861 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:09.305105 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:09.306146 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:09.306691 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:09.306831 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:09.309962 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:09.309971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:09.309990 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:09.310029 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:09.312125 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:09.312721 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:09.312727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:09.312733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:09.312861 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:09.313244 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:09.313247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:09.313249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:09.313811 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:09.314017 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:09.314448 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 00:55:09.314860 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":135,"threads_started":1}
I20260927 00:55:09.317344 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4eff57a13b847159044b7175c2fc576. Latest consensus state: current_term: 1 leader_uuid: "f4eff57a13b847159044b7175c2fc576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260927 00:55:09.317641 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:09.317746 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4eff57a13b847159044b7175c2fc576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260927 00:55:09.317884 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:09.318109 12695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:09.318431 12678 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:55:09.325320 12695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:09.333505 12695 catalog_manager.cc:1347] Generated new cluster ID: 3cca132c9fbb4d7c965c3072010d13f4
I20260927 00:55:09.333654 12695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:09.348306 12695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:55:09.349339 12695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:09.363488 12695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Generated new TSK 0
I20260927 00:55:09.364364 12695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:55:10.064899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:10.190263 12723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:10.190554 12723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:10.190598 12723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:10.194965 12723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:10.195046 12723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:10.195122 12723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:55:10.203070 12723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:10.204277 12723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:10.205369 12723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:10.217301 12731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:10.217324 12729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:10.217772 12723 server_base.cc:1034] running on GCE node
W20260927 00:55:10.217265 12728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:10.218515 12723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:10.219342 12723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:10.222734 12723 hybrid_clock.cc:648] HybridClock initialized: now 1790470510222690 us; error 68 us; skew 500 ppm
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:10.226601 12723 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:55:10.228259 12723 webserver.cc:460] Webserver started at http://127.0.98.1:39105/ using document root <none> and password file <none>
I20260927 00:55:10.229272 12723 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:10.229403 12723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:10.229727 12723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:10.247813 12723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "489236ab2342419bb151a059aac9b3dc"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "5a4e50308855a1cbf01e4a1328cb4ef6"
server_key_iv: "32c5b5d6e94e3ad99e686edbb7e82a5a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:10.248725 12723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "489236ab2342419bb151a059aac9b3dc"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "5a4e50308855a1cbf01e4a1328cb4ef6"
server_key_iv: "32c5b5d6e94e3ad99e686edbb7e82a5a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:10.262986 12723 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260927 00:55:10.288915 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:10.299785 12723 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.000s
I20260927 00:55:10.300143 12723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "489236ab2342419bb151a059aac9b3dc"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "5a4e50308855a1cbf01e4a1328cb4ef6"
server_key_iv: "32c5b5d6e94e3ad99e686edbb7e82a5a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:10.300438 12723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:10.318895 12723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:10.322533 12723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:10.322860 12723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:10.323917 12723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:10.325330 12723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:10.325469 12723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 00:55:10.325624 12723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:10.325726 12723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:10.338004 12723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45431
I20260927 00:55:10.338241 12853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45431 every 8 connection(s)
I20260927 00:55:10.339725 12723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:55:10.347172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 12723
I20260927 00:55:10.347344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:55:10.347718 392 external_mini_cluster.cc:1428] Setting key 70647a1aa27f8be1da34603902e164dc
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:55:10.370693 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:10.344556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.1:34187)
0927 00:55:10.346533 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:10.346612 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:10.348327 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:10.349392 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:10.349407 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:10.349878 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:10.350788 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:10.350814 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.356673 (+ 5859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.356678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:10.358397 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:10.358420 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.359430 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.359437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:10.359777 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:10.360636 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:10.360664 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:10.362958 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:10.366161 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:10.366171 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:10.366197 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:10.366634 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:10.366982 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:10.366985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:10.366988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:10.367280 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:10.367853 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:10.367861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:10.368358 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 00:55:10.368702 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 00:55:10.369234 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"thread_start_us":206,"threads_started":1}
I20260927 00:55:10.372023 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:10.343293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34187 (local address 127.0.98.62:37055)
0927 00:55:10.346178 (+ 2885us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:10.346189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:10.347236 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 00:55:10.348834 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:10.348838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:10.348921 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:10.349021 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:10.355420 (+ 6399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.356409 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.358650 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.359068 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.360265 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:10.363443 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:10.363448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:10.363452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:10.363491 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:10.365869 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:10.366782 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:10.366786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:10.366788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:10.366881 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:10.367402 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:10.367405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:10.367407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:10.367723 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:10.371515 (+ 3792us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:10.371684 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 00:55:10.371864 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6142,"server-negotiator.queue_time_us":2743,"thread_start_us":146,"threads_started":1}
I20260927 00:55:10.373550 12854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260927 00:55:10.373893 12854 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:10.374709 12854 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:10.377475 12642 ts_manager.cc:194] Registered new tserver with Master: 489236ab2342419bb151a059aac9b3dc (127.0.98.1:45431)
I20260927 00:55:10.379276 12642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:55:10.391678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:55:10.571287 12862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:10.571820 12862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:10.571983 12862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:10.578685 12862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:10.578801 12862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:10.578909 12862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:55:10.587854 12862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:10.589766 12862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:10.591570 12862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:10.606729 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:10.606753 12871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:10.606719 12868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:10.607381 12862 server_base.cc:1034] running on GCE node
I20260927 00:55:10.608137 12862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:10.609213 12862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:10.610452 12862 hybrid_clock.cc:648] HybridClock initialized: now 1790470510610426 us; error 46 us; skew 500 ppm
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:10.614697 12862 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:55:10.616200 12862 webserver.cc:460] Webserver started at http://127.0.98.2:36885/ using document root <none> and password file <none>
I20260927 00:55:10.617100 12862 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:10.617304 12862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:10.617587 12862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:10.640348 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a329864471349418e810dbe8807d424"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "81414a04ed651b9c58d236b8e6f47c2f"
server_key_iv: "bb92e3ec84857c3d036d3aada003f184"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:10.642480 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a329864471349418e810dbe8807d424"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "81414a04ed651b9c58d236b8e6f47c2f"
server_key_iv: "bb92e3ec84857c3d036d3aada003f184"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:10.648365 12862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:55:10.668931 12878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:10.673653 12862 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260927 00:55:10.674003 12862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "2a329864471349418e810dbe8807d424"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "81414a04ed651b9c58d236b8e6f47c2f"
server_key_iv: "bb92e3ec84857c3d036d3aada003f184"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:10.674316 12862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:10.694013 12862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:10.698225 12862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:10.698446 12862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:10.699229 12862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:10.700274 12862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:10.700325 12862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:10.700384 12862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:10.700413 12862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:10.713253 12862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44163
I20260927 00:55:10.713280 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44163 every 8 connection(s)
I20260927 00:55:10.715440 12862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:55:10.722584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 12862
I20260927 00:55:10.722731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:55:10.723095 392 external_mini_cluster.cc:1428] Setting key ab6b602ec74f31b672f81c92ccde5605
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:10.735771 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:10.718425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58159 (local address 127.0.98.62:37055)
0927 00:55:10.718659 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:10.718664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:10.719965 (+ 1301us) server_negotiation.cc:407] Connection header received
0927 00:55:10.721233 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:10.721237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:10.721304 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:10.721384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:10.722850 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.723680 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.725489 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.725821 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.725894 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:10.729197 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:10.729202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:10.729205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:10.729237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:10.731026 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:10.731795 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:10.731799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:10.731801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:10.731918 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:10.732382 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:10.732386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:10.732388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:10.732657 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:10.732740 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:10.735367 (+ 2627us) server_negotiation.cc:299] Negotiation successful
0927 00:55:10.735566 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 00:55:10.737674 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:10.718953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.2:58159)
0927 00:55:10.719798 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:10.719856 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:10.720997 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:10.721600 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:10.721609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:10.721926 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:10.722667 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:10.722691 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.723922 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.723925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:10.725345 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:10.725359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:10.726116 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:10.726119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:10.726328 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:10.727103 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:10.727126 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:10.728972 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:10.731199 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:10.731206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:10.731226 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:10.731671 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:10.732043 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:10.732046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:10.732048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:10.732268 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:10.732812 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:10.732820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:10.733266 (+ 446us) client_negotiation.cc:769] Sending connection context
0927 00:55:10.736066 (+ 2800us) client_negotiation.cc:240] Negotiation successful
0927 00:55:10.736516 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":209,"threads_started":1}
I20260927 00:55:10.740527 12992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260927 00:55:10.740908 12992 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:10.741652 12992 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:10.743074 12642 ts_manager.cc:194] Registered new tserver with Master: 2a329864471349418e810dbe8807d424 (127.0.98.2:44163)
I20260927 00:55:10.743963 12642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58159
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:55:10.770762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:10.944661 12997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:10.945062 12997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:10.945137 12997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:10.950881 12997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:10.951068 12997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:10.951229 12997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:55:10.959774 12997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:10.961560 12997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:10.963042 12997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:10.977849 13003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:10.978482 13002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:10.980381 12997 server_base.cc:1034] running on GCE node
Sep 27 00:55:10 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:55:10.980975 13005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:10.981537 12997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:10.982488 12997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:10.994477 12997 hybrid_clock.cc:648] HybridClock initialized: now 1790470510994365 us; error 101 us; skew 500 ppm
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:11.004011 12997 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:55:11.005519 12997 webserver.cc:460] Webserver started at http://127.0.98.3:32811/ using document root <none> and password file <none>
I20260927 00:55:11.006289 12997 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:11.006369 12997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:11.006614 12997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:11.025959 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e5410f541d9452ab62f3fff87190dcd"
format_stamp: "Formatted at 2026-09-27 00:55:11 on dist-test-slave-9fh2"
server_key: "03a9948662e4e9de199ee0efdaffcc43"
server_key_iv: "a694ebb961510b441f2b30315054795a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:11.026881 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e5410f541d9452ab62f3fff87190dcd"
format_stamp: "Formatted at 2026-09-27 00:55:11 on dist-test-slave-9fh2"
server_key: "03a9948662e4e9de199ee0efdaffcc43"
server_key_iv: "a694ebb961510b441f2b30315054795a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:11.039460 12997 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.001s sys 0.003s
I20260927 00:55:11.061467 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.063612 12997 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:55:11.063774 12997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "2e5410f541d9452ab62f3fff87190dcd"
format_stamp: "Formatted at 2026-09-27 00:55:11 on dist-test-slave-9fh2"
server_key: "03a9948662e4e9de199ee0efdaffcc43"
server_key_iv: "a694ebb961510b441f2b30315054795a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:11.063930 12997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:11.091704 12997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:11.096261 12997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:11.096491 12997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:11.097304 12997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:11.098599 12997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:11.098668 12997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.098730 12997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:11.098801 12997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 00:55:11.111644 12997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41427
I20260927 00:55:11.111692 13125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41427 every 8 connection(s)
I20260927 00:55:11.112857 12997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:55:11.113575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 12997
I20260927 00:55:11.113714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:55:11.114077 392 external_mini_cluster.cc:1428] Setting key 2983beac48cec3f433b4cac5f0d5e669
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:11.129956 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:11.115283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33161 (local address 127.0.98.62:37055)
0927 00:55:11.115428 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:11.115432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:11.116618 (+ 1186us) server_negotiation.cc:407] Connection header received
0927 00:55:11.117976 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:11.117980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:11.118050 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:11.118149 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:11.119428 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.120274 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.121710 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.122067 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.122179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:11.125243 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:11.125247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:11.125250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:11.125283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:11.127105 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:11.127950 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:11.127953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:11.127955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:11.128054 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:11.128448 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:11.128452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:11.128454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:11.128727 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:11.128820 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:11.129598 (+ 778us) server_negotiation.cc:299] Negotiation successful
0927 00:55:11.129760 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:55:11.130970 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:11.115854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.3:33161)
0927 00:55:11.116418 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:11.116519 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:11.117735 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:11.118308 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:11.118314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:11.118628 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:11.119280 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:11.119297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.120442 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.120446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:11.121576 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:11.121589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.122253 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.122257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:11.122437 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:11.123172 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:11.123197 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:11.125013 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:11.127329 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:11.127337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:11.127358 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:11.127791 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:11.128141 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:11.128145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:11.128147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:11.128346 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:11.128819 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:11.128828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:11.129267 (+ 439us) client_negotiation.cc:769] Sending connection context
0927 00:55:11.129511 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 00:55:11.129916 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":145,"threads_started":1}
I20260927 00:55:11.132375 13126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260927 00:55:11.132825 13126 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:11.133582 13126 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:11.136009 12642 ts_manager.cc:194] Registered new tserver with Master: 2e5410f541d9452ab62f3fff87190dcd (127.0.98.3:41427)
I20260927 00:55:11.136654 12642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33161
I20260927 00:55:11.141081 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:55:11.141781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 12723
I20260927 00:55:11.147619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 12862
I20260927 00:55:11.153379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 12997
I20260927 00:55:11.159430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 12610
I20260927 00:55:11.168570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--webserver_interface=127.0.98.62
--webserver_port=38305
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055
--ranger_config_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:11.303071 13129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:11.303500 13129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:11.303546 13129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:11.307994 13129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:11.308081 13129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:11.308101 13129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:11.308120 13129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:11.308141 13129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:11.314548 13129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055
--ranger_config_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:11.316097 13129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:11.317414 13129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:11.327819 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:11.327880 13137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:11.327832 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:11.328240 13129 server_base.cc:1034] running on GCE node
I20260927 00:55:11.329028 13129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:11.330523 13129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:11.331729 13129 hybrid_clock.cc:648] HybridClock initialized: now 1790470511331689 us; error 60 us; skew 500 ppm
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:11.335192 13129 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:11.336431 13129 webserver.cc:460] Webserver started at http://127.0.98.62:38305/ using document root <none> and password file <none>
I20260927 00:55:11.337103 13129 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:11.337189 13129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:11.395231 13129 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:55:11.398828 13145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.400939 13129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:55:11.401157 13129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "f4eff57a13b847159044b7175c2fc576"
format_stamp: "Formatted at 2026-09-27 00:55:09 on dist-test-slave-9fh2"
server_key: "88fbc178511d3acebd840a3e07aad2e8"
server_key_iv: "9e2d91886013150af7785629c731b891"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:11.402004 13129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:11.420037 13129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:11.423659 13129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:11.423887 13129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:11.433185 13129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37055
I20260927 00:55:11.433208 13197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37055 every 8 connection(s)
I20260927 00:55:11.434592 13129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:11.437827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13129
I20260927 00:55:11.438349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45431
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37055
--webserver_port=39105
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:55:11.439644 13198 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:55:11.441844 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Bootstrap starting.
I20260927 00:55:11.458412 13198 log.cc:824] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:11.465211 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:55:11.465582 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Bootstrap complete.
I20260927 00:55:11.469619 13198 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:11.470039 13198 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eff57a13b847159044b7175c2fc576, State: Initialized, Role: FOLLOWER
I20260927 00:55:11.470537 13198 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:11.470675 13198 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:11.470741 13198 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:11.470835 13198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:55:11.472946 13198 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:11.473271 13198 leader_election.cc:302] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4eff57a13b847159044b7175c2fc576; no voters:
I20260927 00:55:11.473585 13198 leader_election.cc:288] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:55:11.473897 13202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:55:11.474334 13202 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [term 2 LEADER]: Becoming Leader. State: Replica: f4eff57a13b847159044b7175c2fc576, State: Running, Role: LEADER
I20260927 00:55:11.474880 13198 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:11.474830 13202 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } }
I20260927 00:55:11.476352 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f4eff57a13b847159044b7175c2fc576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260927 00:55:11.476495 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:11.476907 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4eff57a13b847159044b7175c2fc576. Latest consensus state: current_term: 2 leader_uuid: "f4eff57a13b847159044b7175c2fc576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eff57a13b847159044b7175c2fc576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } }
I20260927 00:55:11.477154 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:11.477308 13198 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:55:11.478166 13213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:11.484844 13213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:11.485613 13213 catalog_manager.cc:1259] Loaded cluster ID: 3cca132c9fbb4d7c965c3072010d13f4
I20260927 00:55:11.485803 13213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:11.488818 13213 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:11.489856 13213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Loaded TSK: 0
I20260927 00:55:11.491629 13213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:55:11.592947 13200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:11.593331 13200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:11.593397 13200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:11.599519 13200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:11.599615 13200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:11.599701 13200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:55:11.607340 13200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39105
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:11.608645 13200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:11.609987 13200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:11.623200 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:11.623654 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:11.623935 13200 server_base.cc:1034] running on GCE node
W20260927 00:55:11.624224 13235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:11.624938 13200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:11.625988 13200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:11.627233 13200 hybrid_clock.cc:648] HybridClock initialized: now 1790470511627202 us; error 43 us; skew 500 ppm
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:11.631340 13200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:55:11.632682 13200 webserver.cc:460] Webserver started at http://127.0.98.1:39105/ using document root <none> and password file <none>
I20260927 00:55:11.633481 13200 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:11.633654 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:11.660055 13200 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:55:11.663110 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.664860 13200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:55:11.665005 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "489236ab2342419bb151a059aac9b3dc"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "5a4e50308855a1cbf01e4a1328cb4ef6"
server_key_iv: "32c5b5d6e94e3ad99e686edbb7e82a5a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:11.665704 13200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:11.686930 13200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:11.691217 13200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:11.691552 13200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:11.692454 13200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:11.693830 13200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:11.693933 13200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 00:55:11.694028 13200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:11.694083 13200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.706580 13200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45431
I20260927 00:55:11.706600 13359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45431 every 8 connection(s)
I20260927 00:55:11.708218 13200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:55:11.715410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13200
I20260927 00:55:11.715993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44163
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37055
--webserver_port=36885
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:11.739944 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:11.712455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56007 (local address 127.0.98.62:37055)
0927 00:55:11.712995 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:11.713020 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:11.717964 (+ 4944us) server_negotiation.cc:407] Connection header received
0927 00:55:11.718341 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:11.718352 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:11.718725 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:11.719069 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:11.720590 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.722110 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.723880 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.724642 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.724805 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:11.731829 (+ 7024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:11.731850 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:11.731877 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:11.731937 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:11.735397 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:11.736288 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:11.736295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:11.736300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:11.736457 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:11.736978 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:11.736981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:11.736983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:11.737558 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:11.737686 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:11.738491 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 00:55:11.738852 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":166,"threads_started":1}
I20260927 00:55:11.740839 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:11.712537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.1:56007)
0927 00:55:11.716127 (+ 3590us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:11.716216 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:11.717513 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:11.719210 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:11.719220 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:11.719650 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:11.720354 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:11.720375 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.722283 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.722287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:11.723722 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:11.723737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:11.725121 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:11.725126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:11.725324 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:11.726132 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:11.726156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:11.731370 (+ 5214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:11.735663 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:11.735672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:11.735690 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:11.736126 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:11.736621 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:11.736624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:11.736626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:11.736819 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:11.737759 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:11.737766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:11.738121 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 00:55:11.739189 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0927 00:55:11.739708 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3279,"thread_start_us":169,"threads_started":1}
I20260927 00:55:11.743885 13360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260927 00:55:11.744161 13360 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:11.744791 13360 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
W20260927 00:55:11.747210 13360 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:55:11.747336 13360 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:11.747520 13360 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:11.748481 13360 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:11.748704 13360 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
W20260927 00:55:11.749460 13360 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:55:11.873255 13364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:11.873597 13364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:11.873646 13364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:11.878600 13364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:11.878692 13364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:11.878767 13364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:55:11.885308 13364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36885
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:11.886935 13364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:11.888367 13364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:11.900741 13372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:11.901635 13369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:11.901907 13370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:11.902489 13364 server_base.cc:1034] running on GCE node
I20260927 00:55:11.903183 13364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:11.904186 13364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:11.905507 13364 hybrid_clock.cc:648] HybridClock initialized: now 1790470511905470 us; error 59 us; skew 500 ppm
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:11.909487 13364 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:55:11.910941 13364 webserver.cc:460] Webserver started at http://127.0.98.2:36885/ using document root <none> and password file <none>
I20260927 00:55:11.911893 13364 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:11.912072 13364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:55:11 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:11.954679 13364 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 00:55:11.959797 13380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.961771 13364 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260927 00:55:11.961954 13364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "2a329864471349418e810dbe8807d424"
format_stamp: "Formatted at 2026-09-27 00:55:10 on dist-test-slave-9fh2"
server_key: "81414a04ed651b9c58d236b8e6f47c2f"
server_key_iv: "bb92e3ec84857c3d036d3aada003f184"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:11.962707 13364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:11.984421 13364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:11.990806 13364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:11.991070 13364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:11.992070 13364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:11.993424 13364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:11.993497 13364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:11.993561 13364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:11.993594 13364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:12.010316 13364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44163
I20260927 00:55:12.012789 13493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44163 every 8 connection(s)
I20260927 00:55:12.012780 13364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:55:12.018349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13364
I20260927 00:55:12.018898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41427
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37055
--webserver_port=32811
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:55:12 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:12.033631 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:12.017649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47707 (local address 127.0.98.62:37055)
0927 00:55:12.017804 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:12.017808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:12.017832 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:55:12.019110 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:12.019115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:12.019251 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:12.019322 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:12.021564 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.022462 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.024495 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.024880 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.024974 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:12.028801 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:12.028808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:12.028812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:12.028860 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:12.030522 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:12.031294 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:12.031298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:12.031300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:12.031396 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:12.031894 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:12.031897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:12.031899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:12.032180 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:12.032267 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:12.033185 (+ 918us) server_negotiation.cc:299] Negotiation successful
0927 00:55:12.033376 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:55:12.036078 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:12.016386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.2:47707)
0927 00:55:12.017206 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:12.017320 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:12.018733 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:12.019954 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:12.019968 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:12.020424 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:12.021311 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:12.021336 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.022657 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.022661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:12.024329 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:12.024345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.025277 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.025282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:12.025538 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:12.026391 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:12.026417 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:12.028500 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:12.030671 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:12.030679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:12.030707 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:12.031138 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:12.031507 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:12.031511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:12.031512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:12.031766 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:12.032338 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:12.032347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:12.032835 (+ 488us) client_negotiation.cc:769] Sending connection context
0927 00:55:12.033971 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0927 00:55:12.034475 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":196,"threads_started":1}
I20260927 00:55:12.037959 13495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260927 00:55:12.038323 13495 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.039179 13495 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
W20260927 00:55:12.041514 13495 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:55:12.041693 13495 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.041960 13495 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:12.043068 13495 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.043357 13495 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
W20260927 00:55:12.044313 13495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:55:12.187948 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:12.188439 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:12.188544 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:12.194626 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:12.194785 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:12.194926 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:55:12.202926 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32811
--tserver_master_addrs=127.0.98.62:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:12.204450 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:12.205910 13498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:12.219699 13506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:12.219935 13503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:12.219985 13504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:12.220777 13498 server_base.cc:1034] running on GCE node
I20260927 00:55:12.221516 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:12.222820 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:12.224086 13498 hybrid_clock.cc:648] HybridClock initialized: now 1790470512224049 us; error 94 us; skew 500 ppm
Sep 27 00:55:12 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:12.228096 13498 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:55:12.229331 13498 webserver.cc:460] Webserver started at http://127.0.98.3:32811/ using document root <none> and password file <none>
I20260927 00:55:12.230012 13498 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:12.230105 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:55:12 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:12.260795 13498 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 00:55:12.273027 13514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:12.276662 13498 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260927 00:55:12.276870 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "2e5410f541d9452ab62f3fff87190dcd"
format_stamp: "Formatted at 2026-09-27 00:55:11 on dist-test-slave-9fh2"
server_key: "03a9948662e4e9de199ee0efdaffcc43"
server_key_iv: "a694ebb961510b441f2b30315054795a"
server_key_version: "kuduclusterkey@0"
I20260927 00:55:12.277887 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:12.298015 13498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:12.301416 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:12.301630 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:12.302343 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:12.303437 13498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:12.303552 13498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:12.303658 13498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:12.303720 13498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:12.316093 13498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41427
I20260927 00:55:12.317907 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:55:12.325976 13627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41427 every 8 connection(s)
I20260927 00:55:12.326692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13498
Sep 27 00:55:12 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:12.345398 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:12.328101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.62:37055)
0927 00:55:12.328283 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:12.328288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:12.334352 (+ 6064us) server_negotiation.cc:407] Connection header received
0927 00:55:12.334623 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:12.334626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:12.334688 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:12.334746 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:12.336071 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.336920 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.338324 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.338799 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.338878 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:12.341164 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:12.341169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:12.341172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:12.341239 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:12.343116 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:12.343683 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:12.343688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:12.343690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:12.343782 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:12.344066 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:12.344070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:12.344072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:12.344317 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:12.344568 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:12.345022 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 00:55:12.345187 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:55:12.346134 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:12.329342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50733 (local address 127.0.98.62:37055)
0927 00:55:12.332985 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:12.332990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:12.333017 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:55:12.333069 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:12.333072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:12.333125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:12.333208 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:12.334772 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.335622 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.337292 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.337638 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.337729 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:12.340991 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:12.340995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:12.340999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:12.341031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:12.343373 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:12.343995 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:12.343999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:12.344001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:12.344087 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:12.344452 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:12.344455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:12.344457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:12.344683 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:12.344764 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:12.345796 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0927 00:55:12.345943 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3542,"thread_start_us":129,"threads_started":1}
I20260927 00:55:12.348275 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:12.330045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.98.3:50733)
0927 00:55:12.331425 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:12.331484 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:12.332536 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:12.333533 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:12.333542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:12.333910 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:12.334595 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:12.334614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.335765 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.335769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:12.337131 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:12.337143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:12.337920 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:12.337924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:12.338111 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:12.338864 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:12.338886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:12.340782 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:12.343518 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:12.343524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:12.343542 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:12.343884 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:12.344172 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:12.344175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:12.344177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:12.344345 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:12.344815 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:12.344821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:12.345159 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 00:55:12.345356 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:55:12.347139 (+ 1783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"thread_start_us":194,"threads_started":1}
I20260927 00:55:12.349517 13628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37055
I20260927 00:55:12.350015 13628 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.350919 13628 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
W20260927 00:55:12.353238 13628 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:55:12.353477 13628 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.353770 13628 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:12.354872 13628 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.355113 13628 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
W20260927 00:55:12.356190 13628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:55:12.749747 13360 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:12.750077 13360 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:12.751320 13162 ts_manager.cc:194] Registered new tserver with Master: 489236ab2342419bb151a059aac9b3dc (127.0.98.1:45431)
I20260927 00:55:12.752859 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56007
Sep 27 00:55:12 dist-test-slave-9fh2 krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:13.044622 13495 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:13.044988 13495 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:13.046204 13162 ts_manager.cc:194] Registered new tserver with Master: 2a329864471349418e810dbe8807d424 (127.0.98.2:44163)
I20260927 00:55:13.047204 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47707
Sep 27 00:55:13 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:13.356564 13628 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:13.357030 13628 heartbeater.cc:511] Master 127.0.98.62:37055 requested a full tablet report, sending...
I20260927 00:55:13.358409 13162 ts_manager.cc:194] Registered new tserver with Master: 2e5410f541d9452ab62f3fff87190dcd (127.0.98.3:41427)
I20260927 00:55:13.359426 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50733
I20260927 00:55:13.364765 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:55:13.377993 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.366582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.98.62:37055)
0927 00:55:13.366964 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.366969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.367030 (+ 61us) server_negotiation.cc:407] Connection header received
0927 00:55:13.367277 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.367281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.367344 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.367464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:13.368486 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.369370 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.370727 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.371104 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.371261 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.373471 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:13.373475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:13.373478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:13.373507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:13.375581 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:13.376241 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:13.376248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:13.376250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:13.376411 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:13.376729 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:13.376733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:13.376735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:13.377134 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:13.377309 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.377586 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.377772 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":99,"threads_started":1}
I20260927 00:55:13.383764 13162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:55:13.390074 13162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:55:13.415477 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.403960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45431 (local address 127.0.0.1:52264)
0927 00:55:13.407298 (+ 3338us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.407359 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:13.408138 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:13.408912 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:13.408918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:13.408941 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:13.409086 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.409094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.411424 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.411428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.412945 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.412958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.414580 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.414584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.414712 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.414959 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 00:55:13.415104 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:55:13.415273 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3122,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260927 00:55:13.416738 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.403967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44163 (local address 127.0.0.1:44608)
0927 00:55:13.409288 (+ 5321us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.409303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:13.409381 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:13.409915 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:13.409918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:13.409943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:13.410089 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.410097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.411717 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.411721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.414170 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.414184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.415883 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.415886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.415985 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.416010 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:55:13.416104 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:55:13.416564 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5260,"mutex_wait_us":4,"thread_start_us":170,"threads_started":1}
I20260927 00:55:13.417520 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.405008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.0.98.2:44163)
0927 00:55:13.407829 (+ 2821us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.407835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.409485 (+ 1650us) server_negotiation.cc:407] Connection header received
0927 00:55:13.409569 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.409574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.409749 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.409890 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:13.410262 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.411562 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.414357 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.415509 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.415632 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.415809 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.417148 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.417351 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2719,"thread_start_us":150,"threads_started":1}
I20260927 00:55:13.420109 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.405437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52264 (local address 127.0.98.1:45431)
0927 00:55:13.407994 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.408001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.408025 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:55:13.408342 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.408348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.408585 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.408729 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:13.410052 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.411269 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.413100 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.414198 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.414339 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.414505 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.419814 (+ 5309us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.419945 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2465,"thread_start_us":119,"threads_started":1}
I20260927 00:55:13.422196 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1428a93c9a4969be55be10cbb8342b (DEFAULT_TABLE table=test-table [id=fe62ce9a15ef45418fb5fc16e2cc29b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:13.423906 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1428a93c9a4969be55be10cbb8342b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:13.426283 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1428a93c9a4969be55be10cbb8342b (DEFAULT_TABLE table=test-table [id=fe62ce9a15ef45418fb5fc16e2cc29b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:13.428381 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1428a93c9a4969be55be10cbb8342b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:13.439885 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.406192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41427 (local address 127.0.0.1:60270)
0927 00:55:13.416323 (+ 10131us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.416337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:13.417537 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:13.419569 (+ 2032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:13.419572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:13.419589 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:13.419672 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.419677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.422611 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.422614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.437623 (+ 15009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.437644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.439376 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.439381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.439490 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.439520 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:55:13.439599 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:55:13.439709 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10058,"thread_start_us":96,"threads_started":1}
I20260927 00:55:13.440436 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.406630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.0.98.3:41427)
0927 00:55:13.407643 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.407652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.416414 (+ 8762us) server_negotiation.cc:407] Connection header received
0927 00:55:13.418987 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.418994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.419243 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.419378 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:13.420405 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.422485 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.437849 (+ 15364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.438901 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.439072 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.439298 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.440147 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.440287 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":72,"threads_started":1}
I20260927 00:55:13.444049 13562 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1428a93c9a4969be55be10cbb8342b (DEFAULT_TABLE table=test-table [id=fe62ce9a15ef45418fb5fc16e2cc29b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:13.445269 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1428a93c9a4969be55be10cbb8342b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:13.447613 13661 tablet_bootstrap.cc:492] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: Bootstrap starting.
I20260927 00:55:13.450086 13661 tablet_bootstrap.cc:654] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:13.450964 13661 log.cc:824] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:13.451774 13662 tablet_bootstrap.cc:492] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc: Bootstrap starting.
I20260927 00:55:13.453148 13661 tablet_bootstrap.cc:492] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: No bootstrap required, opened a new log
I20260927 00:55:13.453377 13661 ts_tablet_manager.cc:1397] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:55:13.455700 13662 tablet_bootstrap.cc:654] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:13.456552 13665 tablet_bootstrap.cc:492] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: Bootstrap starting.
I20260927 00:55:13.456674 13662 log.cc:824] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:13.457531 13661 raft_consensus.cc:348] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.457791 13661 raft_consensus.cc:374] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:13.457841 13661 raft_consensus.cc:729] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a329864471349418e810dbe8807d424, State: Initialized, Role: FOLLOWER
I20260927 00:55:13.458415 13661 consensus_queue.cc:260] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.459314 13495 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260927 00:55:13.459465 13661 ts_tablet_manager.cc:1428] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:55:13.459609 13665 tablet_bootstrap.cc:654] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:13.460397 13665 log.cc:824] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:13.463225 13665 tablet_bootstrap.cc:492] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: No bootstrap required, opened a new log
I20260927 00:55:13.463492 13665 ts_tablet_manager.cc:1397] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:55:13.465498 13662 tablet_bootstrap.cc:492] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc: No bootstrap required, opened a new log
I20260927 00:55:13.465754 13662 ts_tablet_manager.cc:1397] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260927 00:55:13.467759 13665 raft_consensus.cc:348] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.468189 13665 raft_consensus.cc:374] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:13.468277 13665 raft_consensus.cc:729] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5410f541d9452ab62f3fff87190dcd, State: Initialized, Role: FOLLOWER
I20260927 00:55:13.468961 13665 consensus_queue.cc:260] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.470505 13662 raft_consensus.cc:348] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.470819 13662 raft_consensus.cc:374] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:13.470917 13662 raft_consensus.cc:729] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489236ab2342419bb151a059aac9b3dc, State: Initialized, Role: FOLLOWER
I20260927 00:55:13.471356 13665 ts_tablet_manager.cc:1428] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:55:13.471467 13662 consensus_queue.cc:260] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.471722 13628 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260927 00:55:13.472527 13662 ts_tablet_manager.cc:1428] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260927 00:55:13.472863 13360 heartbeater.cc:503] Master 127.0.98.62:37055 was elected leader, sending a full tablet report...
I20260927 00:55:13.479350 13670 raft_consensus.cc:482] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:13.479753 13670 raft_consensus.cc:504] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.483325 13670 leader_election.cc:288] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a329864471349418e810dbe8807d424 (127.0.98.2:44163), 2e5410f541d9452ab62f3fff87190dcd (127.0.98.3:41427)
Sep 27 00:55:13 dist-test-slave-9fh2 krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:55:13.490931 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.481745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44163 (local address 127.0.98.1:58485)
0927 00:55:13.482321 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.482341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:13.482549 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:13.482964 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:13.482967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:13.482986 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:13.483134 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.483140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.484268 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.484272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.487516 (+ 3244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.487524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.490368 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.490372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.490487 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.490514 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:55:13.490598 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:55:13.490700 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":174,"thread_start_us":156,"threads_started":1}
I20260927 00:55:13.490950 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.482026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58485 (local address 127.0.98.2:44163)
0927 00:55:13.482260 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.482265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.482437 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:55:13.482690 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.482694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.482749 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.482795 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:13.483290 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.484063 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.488873 (+ 4810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.489586 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.489682 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.489755 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.490658 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.490797 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:55:13.491528 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.483101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48805 (local address 127.0.98.3:41427)
0927 00:55:13.483262 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.483267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.483858 (+ 591us) server_negotiation.cc:407] Connection header received
0927 00:55:13.484098 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.484102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.484167 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.484224 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:13.484746 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.485595 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.487655 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.488312 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.488401 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.488488 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.491270 (+ 2782us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.491393 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:55:13.491987 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.482631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41427 (local address 127.0.98.1:48805)
0927 00:55:13.483747 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.483764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:13.483946 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:13.484434 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:13.484437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:13.484456 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:13.484614 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.484621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.485839 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.485843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.487340 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:13.487351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.488593 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.488596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:13.488703 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.488728 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:55:13.491749 (+ 3021us) client_negotiation.cc:240] Negotiation successful
0927 00:55:13.491836 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":10,"thread_start_us":105,"threads_started":1}
I20260927 00:55:13.491979 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1428a93c9a4969be55be10cbb8342b" candidate_uuid: "489236ab2342419bb151a059aac9b3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a329864471349418e810dbe8807d424" is_pre_election: true
I20260927 00:55:13.492323 13433 raft_consensus.cc:2454] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 489236ab2342419bb151a059aac9b3dc in term 0.
I20260927 00:55:13.492694 13582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1428a93c9a4969be55be10cbb8342b" candidate_uuid: "489236ab2342419bb151a059aac9b3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5410f541d9452ab62f3fff87190dcd" is_pre_election: true
I20260927 00:55:13.493075 13582 raft_consensus.cc:2454] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 489236ab2342419bb151a059aac9b3dc in term 0.
I20260927 00:55:13.493134 13248 leader_election.cc:302] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a329864471349418e810dbe8807d424, 489236ab2342419bb151a059aac9b3dc; no voters:
I20260927 00:55:13.493682 13670 raft_consensus.cc:2781] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:55:13.493767 13670 raft_consensus.cc:482] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:55:13.493803 13670 raft_consensus.cc:3037] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:13.495123 13670 raft_consensus.cc:504] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.496022 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1428a93c9a4969be55be10cbb8342b" candidate_uuid: "489236ab2342419bb151a059aac9b3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a329864471349418e810dbe8807d424"
I20260927 00:55:13.496150 13433 raft_consensus.cc:3037] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:13.497589 13433 raft_consensus.cc:2454] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489236ab2342419bb151a059aac9b3dc in term 1.
I20260927 00:55:13.497987 13248 leader_election.cc:302] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a329864471349418e810dbe8807d424, 489236ab2342419bb151a059aac9b3dc; no voters:
I20260927 00:55:13.498912 13582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1428a93c9a4969be55be10cbb8342b" candidate_uuid: "489236ab2342419bb151a059aac9b3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5410f541d9452ab62f3fff87190dcd"
I20260927 00:55:13.499078 13582 raft_consensus.cc:3037] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:13.500550 13582 raft_consensus.cc:2454] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489236ab2342419bb151a059aac9b3dc in term 1.
I20260927 00:55:13.501044 13670 leader_election.cc:288] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [CANDIDATE]: Term 1 election: Requested vote from peers 2a329864471349418e810dbe8807d424 (127.0.98.2:44163), 2e5410f541d9452ab62f3fff87190dcd (127.0.98.3:41427)
I20260927 00:55:13.501248 13670 raft_consensus.cc:2781] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:13.501362 13670 raft_consensus.cc:686] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [term 1 LEADER]: Becoming Leader. State: Replica: 489236ab2342419bb151a059aac9b3dc, State: Running, Role: LEADER
I20260927 00:55:13.501747 13670 consensus_queue.cc:237] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } }
I20260927 00:55:13.505191 13162 catalog_manager.cc:5495] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 489236ab2342419bb151a059aac9b3dc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "489236ab2342419bb151a059aac9b3dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "489236ab2342419bb151a059aac9b3dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 } health_report { overall_health: UNKNOWN } } }
W20260927 00:55:13.516508 13496 tablet.cc:2369] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:55:13.542251 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.536835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.1:45431)
0927 00:55:13.536991 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:13.536994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:13.537006 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:55:13.537117 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:13.537121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:13.537213 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:13.537357 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:55:13.537802 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.538538 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.539773 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:13.540052 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:13.540173 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:13.540653 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:55:13.541406 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:55:13.541522 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:13.542015 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 00:55:13.542082 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:55:13.550513 13582 raft_consensus.cc:1261] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 1 FOLLOWER]: Refusing update from remote peer 489236ab2342419bb151a059aac9b3dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:13.551676 13673 consensus_queue.cc:1035] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5410f541d9452ab62f3fff87190dcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:13.553818 13433 raft_consensus.cc:1261] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 1 FOLLOWER]: Refusing update from remote peer 489236ab2342419bb151a059aac9b3dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:13.554697 13673 consensus_queue.cc:1035] T 6f1428a93c9a4969be55be10cbb8342b P 489236ab2342419bb151a059aac9b3dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a329864471349418e810dbe8807d424" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:13.570221 13677 mvcc.cc:205] Tried to move back new op lower bound from 7333767223491497984 to 7333767223306231808. Current Snapshot: MvccSnapshot[applied={T|T < 7333767223491497984 or (T in {7333767223491497984})}]
I20260927 00:55:13.589375 13158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:13.589581 13158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:13.592128 13678 mvcc.cc:205] Tried to move back new op lower bound from 7333767223491497984 to 7333767223306231808. Current Snapshot: MvccSnapshot[applied={T|T < 7333767223491497984 or (T in {7333767223491497984})}]
I20260927 00:55:13.593961 13158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4eff57a13b847159044b7175c2fc576: Sending DeleteTablet for 3 replicas of tablet 6f1428a93c9a4969be55be10cbb8342b
I20260927 00:55:13.594944 13428 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1428a93c9a4969be55be10cbb8342b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44608
I20260927 00:55:13.596194 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1428a93c9a4969be55be10cbb8342b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52264
I20260927 00:55:13.596724 13562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1428a93c9a4969be55be10cbb8342b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60270
I20260927 00:55:13.598265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13200
I20260927 00:55:13.602588 13688 tablet_replica.cc:330] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: stopping tablet replica
I20260927 00:55:13.602941 13688 raft_consensus.cc:2229] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:13.603360 13688 raft_consensus.cc:2258] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:13.605875 13686 tablet_replica.cc:330] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424: stopping tablet replica
I20260927 00:55:13.607043 13686 raft_consensus.cc:2229] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:13.608086 13688 ts_tablet_manager.cc:1905] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:13.608531 13686 raft_consensus.cc:2258] T 6f1428a93c9a4969be55be10cbb8342b P 2a329864471349418e810dbe8807d424 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:55:13.615640 13146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45431 (error 108)
I20260927 00:55:13.615063 13688 ts_tablet_manager.cc:1918] T 6f1428a93c9a4969be55be10cbb8342b P 2e5410f541d9452ab62f3fff87190dcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:13.616827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13364
I20260927 00:55:13.625501 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45431 (local address 127.0.0.1:52288)
0927 00:55:13.625310 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.625396 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 00:55:13.625837 13146 catalog_manager.cc:4584] TS 489236ab2342419bb151a059aac9b3dc (127.0.98.1:45431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f1428a93c9a4969be55be10cbb8342b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45431: connect: Connection refused (error 111)
I20260927 00:55:13.626724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13498
I20260927 00:55:13.630599 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:13.630286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44163 (local address 127.0.0.1:44616)
0927 00:55:13.630466 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.630514 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 00:55:13.636664 13659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:55:13.636098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41427 (local address 127.0.0.1:60272)
0927 00:55:13.636273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:13.636288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:13.636374 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:13.636552 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:41427: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 00:55:13.637348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13129
I20260927 00:55:13.644189 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:55:13.989137 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:55:13.989264 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:55:14.336714 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:55:14.336 UTC [12195] LOG: received smart shutdown request
2026-09-27 00:55:14.341 UTC [12195] LOG: background worker "logical replication launcher" (PID 12204) exited with exit code 1
2026-09-27 00:55:14.341 UTC [12199] LOG: shutting down
2026-09-27 00:55:14.553 UTC [12195] LOG: database system is shut down
2026-09-27T00:55:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52344 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:55:14 dist-test-slave-9fh2 krb5kdc[13698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:55:14 dist-test-slave-9fh2 krb5kdc[13698](info): set up 2 sockets
Sep 27 00:55:14 dist-test-slave-9fh2 krb5kdc[13698](info): commencing operation
krb5kdc: starting...
W20260927 00:55:15.635525 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:55:15 dist-test-slave-9fh2 krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:55:15.770978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39695,127.0.98.61:43443,127.0.98.60:38163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:15.882957 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:15.883275 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:15.883334 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:15.887468 13711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:15.887547 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:15.887576 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:15.887598 13711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:15.887619 13711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:15.893129 13711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39695,127.0.98.61:43443,127.0.98.60:38163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:15.894246 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:15.895278 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:15.904703 13716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:15.904951 13711 server_base.cc:1034] running on GCE node
W20260927 00:55:15.904816 13717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:15.904703 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:15.905815 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:15.907372 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:15.908571 13711 hybrid_clock.cc:648] HybridClock initialized: now 1790470515908551 us; error 39 us; skew 500 ppm
Sep 27 00:55:15 dist-test-slave-9fh2 krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:15.911991 13711 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:15.913188 13711 webserver.cc:460] Webserver started at http://127.0.98.62:46063/ using document root <none> and password file <none>
I20260927 00:55:15.914103 13711 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:15.914198 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:15.914453 13711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:15.917213 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "85725052bd5f459b92fa2fd158af009a"
format_stamp: "Formatted at 2026-09-27 00:55:15 on dist-test-slave-9fh2"
server_key: "f3715cd1d44e19b79d92712079fba87a"
server_key_iv: "cc1f82386c65f436f049e539ea9bbc30"
server_key_version: "encryptionkey@0"
I20260927 00:55:15.917901 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "85725052bd5f459b92fa2fd158af009a"
format_stamp: "Formatted at 2026-09-27 00:55:15 on dist-test-slave-9fh2"
server_key: "f3715cd1d44e19b79d92712079fba87a"
server_key_iv: "cc1f82386c65f436f049e539ea9bbc30"
server_key_version: "encryptionkey@0"
I20260927 00:55:15.922672 13711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 00:55:15.925916 13726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:15.927654 13711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:55:15.927882 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "85725052bd5f459b92fa2fd158af009a"
format_stamp: "Formatted at 2026-09-27 00:55:15 on dist-test-slave-9fh2"
server_key: "f3715cd1d44e19b79d92712079fba87a"
server_key_iv: "cc1f82386c65f436f049e539ea9bbc30"
server_key_version: "encryptionkey@0"
I20260927 00:55:15.928027 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:15.940171 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:15.944434 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:15.944669 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:15.953208 13711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39695
I20260927 00:55:15.953231 13778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39695 every 8 connection(s)
I20260927 00:55:15.954478 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:15.957713 13779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:15.959064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13711
I20260927 00:55:15.959223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:55:15.959545 392 external_mini_cluster.cc:1428] Setting key d95b76fbfe64339db7b85b0a53d18250
I20260927 00:55:15.962443 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:55:15 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:15.980086 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:15.965349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.0.1:46402)
0927 00:55:15.966048 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:15.966105 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:15.966994 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:15.967690 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:15.967695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:15.968157 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:15.968396 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:15.968411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:15.969628 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:15.969630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:15.970358 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:15.970367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:15.971049 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:15.971053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:15.971274 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:15.971833 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:15.971850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:15.973351 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:15.976294 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:15.976306 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:15.976317 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:15.976681 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:15.977132 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:15.977134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:15.977136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:15.977269 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:15.978266 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:15.978273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:15.978462 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 00:55:15.978629 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 00:55:15.978982 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":78,"thread_start_us":241,"threads_started":1}
I20260927 00:55:15.980089 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:15.965348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.0.98.62:39695)
0927 00:55:15.966048 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:15.966072 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:15.966226 (+ 154us) server_negotiation.cc:407] Connection header received
0927 00:55:15.967312 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:15.967316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:15.967506 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:15.967668 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:15.968536 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:15.969509 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:15.970507 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:15.970898 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:15.971043 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:15.973631 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:15.973642 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:15.973659 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:15.973701 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:15.976094 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:15.976822 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:15.976832 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:15.976834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:15.977027 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:15.977372 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:15.977376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:15.977377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:15.978109 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:15.978287 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:15.978760 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 00:55:15.979097 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":586,"spinlock_wait_cycles":44416,"thread_start_us":221,"threads_started":1}
I20260927 00:55:15.982625 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:55:15.983577 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:15.983188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:42948)
0927 00:55:15.983366 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:15.983484 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 00:55:15.983794 13727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
I20260927 00:55:15.984720 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:15.984402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:42960)
0927 00:55:15.984590 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:15.984650 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 00:55:15.984877 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:55:15.991268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43443
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39695,127.0.98.61:43443,127.0.98.60:38163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:55:16.034143 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } attempt: 1
I20260927 00:55:16.035346 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.034764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:42970)
0927 00:55:16.035057 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.035204 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 00:55:16.036573 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.036198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:42978)
0927 00:55:16.036370 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.036457 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 00:55:16.036906 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
I20260927 00:55:16.105111 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } attempt: 2
I20260927 00:55:16.106241 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.105862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:42988)
0927 00:55:16.106076 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.106158 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 00:55:16.107059 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.106858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:42998)
0927 00:55:16.106959 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.107006 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:55:16.107203 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
W20260927 00:55:16.136487 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:16.136890 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:16.136938 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:16.141620 13786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:16.141721 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:16.141742 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:16.141760 13786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:16.141776 13786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:16.148020 13786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39695,127.0.98.61:43443,127.0.98.60:38163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:16.149626 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:16.151021 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:16.161530 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:16.161455 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:16.161561 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:16.162029 13786 server_base.cc:1034] running on GCE node
I20260927 00:55:16.162878 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:16.164462 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:16.165666 13786 hybrid_clock.cc:648] HybridClock initialized: now 1790470516165641 us; error 48 us; skew 500 ppm
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:16.172905 13786 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:55:16.174273 13786 webserver.cc:460] Webserver started at http://127.0.98.61:39809/ using document root <none> and password file <none>
I20260927 00:55:16.175002 13786 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:16.175082 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:16.175360 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:16.178011 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data/instance:
uuid: "a57c87d10eb745538b6d6b7992fd22fe"
format_stamp: "Formatted at 2026-09-27 00:55:16 on dist-test-slave-9fh2"
server_key: "8f89cea1036b376649d9dfa76364d8c9"
server_key_iv: "adec92d1225e7e09b95bdff6a7669b23"
server_key_version: "encryptionkey@0"
I20260927 00:55:16.178709 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal/instance:
uuid: "a57c87d10eb745538b6d6b7992fd22fe"
format_stamp: "Formatted at 2026-09-27 00:55:16 on dist-test-slave-9fh2"
server_key: "8f89cea1036b376649d9dfa76364d8c9"
server_key_iv: "adec92d1225e7e09b95bdff6a7669b23"
server_key_version: "encryptionkey@0"
I20260927 00:55:16.183858 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:55:16.187341 13801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:16.189154 13786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:55:16.189354 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal
uuid: "a57c87d10eb745538b6d6b7992fd22fe"
format_stamp: "Formatted at 2026-09-27 00:55:16 on dist-test-slave-9fh2"
server_key: "8f89cea1036b376649d9dfa76364d8c9"
server_key_iv: "adec92d1225e7e09b95bdff6a7669b23"
server_key_version: "encryptionkey@0"
I20260927 00:55:16.189504 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:16.198379 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } attempt: 3
I20260927 00:55:16.199507 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.199086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:43014)
0927 00:55:16.199345 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.199426 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 00:55:16.200208 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.200023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:43030)
0927 00:55:16.200115 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.200160 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:55:16.200323 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43443: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43443: connect: Connection refused (error 111)
I20260927 00:55:16.224151 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:16.228587 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:16.228842 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:16.238232 13786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43443
I20260927 00:55:16.238258 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43443 every 8 connection(s)
I20260927 00:55:16.239640 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:55:16.241660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13786
I20260927 00:55:16.241878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-1/wal/instance
I20260927 00:55:16.242183 392 external_mini_cluster.cc:1428] Setting key a5a3e48b29411d4c63f3f58d494ef2e3
I20260927 00:55:16.242940 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:16.246910 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:16.263897 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.249063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.0.98.62:39695)
0927 00:55:16.249288 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.249294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.250069 (+ 775us) server_negotiation.cc:407] Connection header received
0927 00:55:16.251079 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.251083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.251242 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.251357 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.253001 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.253794 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.255716 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.256018 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.256126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.258846 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.258852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.258856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.258910 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.260795 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.261436 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.261440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.261442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.261541 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.261987 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.261991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.261993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.262300 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.262411 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.263210 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.263355 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 00:55:16.264523 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.249347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.0.1:46412)
0927 00:55:16.249860 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.249922 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.250836 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.251661 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.251668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.252067 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.252789 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.252806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.253978 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.253981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.255511 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.255533 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.256209 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.256214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.256391 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.257018 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.257037 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.258547 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.260977 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.260984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.260999 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.261310 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.261677 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.261680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.261682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.261850 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.262527 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.262533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.262854 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.263063 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.263515 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260927 00:55:16.266440 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:55:16.274466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38163
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39695,127.0.98.61:43443,127.0.98.60:38163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:55:16.278224 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.267171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:43036)
0927 00:55:16.267354 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.267383 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.267490 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.268058 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.268061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.268364 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.268490 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.268497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.269833 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.269837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.270519 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.270530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.271257 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.271261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.271352 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.271902 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.271914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.273368 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.276118 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.276123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.276124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.276395 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.276831 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.276834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.276835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.276930 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.277604 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.277607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.277754 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.277892 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.278051 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 00:55:16.278437 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.267288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.61:43443)
0927 00:55:16.267700 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.267709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.267728 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:55:16.267763 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.267767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.267937 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.268058 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.268638 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.269686 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.270630 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.271081 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.271236 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.273611 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.273631 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.273655 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.273701 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.275932 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.276536 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.276545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.276547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.276673 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.277028 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.277032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.277033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.277462 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.277611 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.277988 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.278166 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":173,"threads_started":1}
I20260927 00:55:16.280834 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:55:16.281800 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.281389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47772)
0927 00:55:16.281560 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.281693 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 00:55:16.282063 13805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
I20260927 00:55:16.283027 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.282737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47788)
0927 00:55:16.282873 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.282951 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 00:55:16.283257 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
I20260927 00:55:16.332664 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 1
I20260927 00:55:16.334014 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.333505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47804)
0927 00:55:16.333800 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.333900 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 00:55:16.334883 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.334640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47806)
0927 00:55:16.334766 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.334826 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:55:16.335024 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
I20260927 00:55:16.343505 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } attempt: 4
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:55:16.353888 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.344234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:43050)
0927 00:55:16.344443 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.344460 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.344578 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.344830 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.344833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.345073 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.345209 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.345215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.346036 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.346039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.346625 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.346632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.347671 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.347675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.347799 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.348429 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.348442 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.350056 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.352057 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.352062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.352065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.352399 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.352718 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.352721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.352723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.352831 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.353255 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.353258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.353414 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.353533 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.353703 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 00:55:16.353899 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.344356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.0.98.61:43443)
0927 00:55:16.344578 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.344584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.344604 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:55:16.344669 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.344672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.344723 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.344801 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.345330 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.345920 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.346732 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.347546 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.347652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.350290 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.350294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.350296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.350327 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.351893 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.352521 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.352524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.352525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.352615 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.352931 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.352933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.352934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.353152 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.353234 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.353605 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.353742 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 00:55:16.354977 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:55:16.355964 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.355630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47822)
0927 00:55:16.355816 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.355883 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 00:55:16.356787 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.356568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47824)
0927 00:55:16.356677 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.356723 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:55:16.356911 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
W20260927 00:55:16.397541 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:16.397826 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:16.397891 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:16.401998 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:16.402064 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:16.402091 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:16.402113 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:16.402141 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:16.403273 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 2
I20260927 00:55:16.404340 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.403929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47838)
0927 00:55:16.404142 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.404229 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 00:55:16.405144 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.404940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47852)
0927 00:55:16.405045 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.405092 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:55:16.405264 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
I20260927 00:55:16.407891 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39695,127.0.98.61:43443,127.0.98.60:38163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:16.409008 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:16.410030 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:55:16.416085 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 1
I20260927 00:55:16.417151 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.416750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47854)
0927 00:55:16.416942 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.417037 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 00:55:16.417987 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.417743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47870)
0927 00:55:16.417857 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.417917 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:55:16.418179 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
W20260927 00:55:16.420150 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:16.420167 13866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:16.420197 13867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:16.420390 13860 server_base.cc:1034] running on GCE node
I20260927 00:55:16.421236 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:16.422561 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:16.423787 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790470516423717 us; error 93 us; skew 500 ppm
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:16.426882 13860 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:55:16.427948 13860 webserver.cc:460] Webserver started at http://127.0.98.60:45457/ using document root <none> and password file <none>
I20260927 00:55:16.428532 13860 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:16.428601 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:16.428791 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:16.430917 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data/instance:
uuid: "e33958c0df3d4d03b6f15e92e1e20170"
format_stamp: "Formatted at 2026-09-27 00:55:16 on dist-test-slave-9fh2"
server_key: "12ca7d5f2811ff88a3625b9e9d711816"
server_key_iv: "2007f61b29f83a4f67b7f045b29e9939"
server_key_version: "encryptionkey@0"
I20260927 00:55:16.431502 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal/instance:
uuid: "e33958c0df3d4d03b6f15e92e1e20170"
format_stamp: "Formatted at 2026-09-27 00:55:16 on dist-test-slave-9fh2"
server_key: "12ca7d5f2811ff88a3625b9e9d711816"
server_key_iv: "2007f61b29f83a4f67b7f045b29e9939"
server_key_version: "encryptionkey@0"
I20260927 00:55:16.435849 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:55:16.438961 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:16.440604 13860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:16.440747 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal
uuid: "e33958c0df3d4d03b6f15e92e1e20170"
format_stamp: "Formatted at 2026-09-27 00:55:16 on dist-test-slave-9fh2"
server_key: "12ca7d5f2811ff88a3625b9e9d711816"
server_key_iv: "2007f61b29f83a4f67b7f045b29e9939"
server_key_version: "encryptionkey@0"
I20260927 00:55:16.440857 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:16.485397 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 2
I20260927 00:55:16.486615 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.486176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47882)
0927 00:55:16.486400 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.486493 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 00:55:16.487494 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.487292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47892)
0927 00:55:16.487393 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.487440 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 00:55:16.487722 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
I20260927 00:55:16.496433 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 3
I20260927 00:55:16.497437 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.497030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47908)
0927 00:55:16.497224 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.497322 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 00:55:16.497865 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:16.498292 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.498026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47912)
0927 00:55:16.498184 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.498237 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 00:55:16.498443 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38163: connect: Connection refused (error 111)
I20260927 00:55:16.501184 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:16.501360 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:16.508869 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38163
I20260927 00:55:16.508916 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38163 every 8 connection(s)
I20260927 00:55:16.509907 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:55:16.513489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 13860
I20260927 00:55:16.513608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470348013064-392-0/minicluster-data/master-2/wal/instance
I20260927 00:55:16.513736 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:16.513899 392 external_mini_cluster.cc:1428] Setting key 38e05775023bd5a2894871b4b75b323c
I20260927 00:55:16.520095 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:16.530905 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.515510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.0.98.60:38163)
0927 00:55:16.516014 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.516037 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.516093 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:55:16.517194 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.517222 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.517780 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.518268 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.519006 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.520591 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.521555 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.522242 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.522446 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.525261 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.525270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.525282 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.525337 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.527327 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.527945 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.527950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.527955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.528090 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.528383 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.528387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.528388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.528970 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.529182 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.529514 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.529830 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":132,"threads_started":1}
I20260927 00:55:16.533215 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.522504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.98.62:39695)
0927 00:55:16.522678 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.522682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.523043 (+ 361us) server_negotiation.cc:407] Connection header received
0927 00:55:16.523287 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.523291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.523339 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.523411 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.524218 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.524875 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.526200 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.526509 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.526627 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.529455 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.529459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.529462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.529488 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.531016 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.531616 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.531620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.531622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.531803 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.532132 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.532136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.532138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.532417 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.532540 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.532947 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.533074 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:55:16.533226 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.522366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.0.1:46414)
0927 00:55:16.522904 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.522964 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.523170 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.523484 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.523490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.523900 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.524067 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.524074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.525015 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.525047 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.526071 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.526084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.526655 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.526658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.526804 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.527462 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.527478 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.529308 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.531183 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.531190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.531196 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.531471 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.531918 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.531921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.531923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.532019 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.532563 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.532568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.532686 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.532828 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.533035 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":117,"threads_started":1}
I20260927 00:55:16.534667 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:16.543242 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.535286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43443 (local address 127.0.0.1:43064)
0927 00:55:16.535444 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.535458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.535526 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.535809 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.535812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.535978 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.536071 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.536077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.536779 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.536783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.537350 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.537358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.537770 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.537773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.537865 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.538323 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.538335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.539781 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.541626 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.541629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.541632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.541877 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.542147 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.542150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.542153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.542238 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.542630 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.542633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.542741 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.542831 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.543003 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 00:55:16.543242 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.535455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.0.98.61:43443)
0927 00:55:16.535625 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.535630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.535644 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:55:16.535682 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.535684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.535721 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.535784 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.536172 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.536686 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.537452 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.537678 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.537762 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.539964 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.539968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.539970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.539999 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.541509 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.541970 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.541974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.541976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.542055 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.542326 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.542329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.542331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.542538 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.542625 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.542894 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.543007 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:55:16.544138 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.534505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.0.98.62:39695)
0927 00:55:16.534606 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.534610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.534737 (+ 127us) server_negotiation.cc:407] Connection header received
0927 00:55:16.534940 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.534944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.534997 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.535070 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.535886 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.536434 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.537226 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.537470 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.537560 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.540421 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.540425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.540427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.540453 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.541922 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.542420 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.542422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.542424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.542499 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.542746 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.542749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.542751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.542944 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.543191 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.543900 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.544020 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:55:16.544178 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:55:16.552306 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.544809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.0.98.60:38163)
0927 00:55:16.544951 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.544956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.544972 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:55:16.545077 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.545079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.545127 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.545204 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.545625 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.546193 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.546912 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.547174 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.547285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.549372 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.549376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.549379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.549402 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.550772 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.551141 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.551170 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.551172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.551257 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.551485 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.551488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.551490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.551680 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.551766 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.551987 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.552163 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":55}
I20260927 00:55:16.552306 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.544760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47918)
0927 00:55:16.544888 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.544903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.544981 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.545212 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.545214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.545356 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.545471 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.545479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.546286 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.546289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.546801 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.546808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.547260 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.547263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.547332 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.547781 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.547792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.549241 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.550873 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.550876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.550879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.551060 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.551331 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.551334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.551336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.551412 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.551767 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.551770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.551851 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.551939 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.552098 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:55:16.553979 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.545515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.0.98.61:43443)
0927 00:55:16.545642 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.545647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.545661 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:55:16.545698 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.545701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.545741 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.545804 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.546408 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.546903 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.547672 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.547892 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.547977 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.550308 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.550312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.550314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.550338 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.551854 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.552245 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.552249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.552250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.552327 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.553045 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.553048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.553050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.553293 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.553528 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.553682 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.553816 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:55:16.556814 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170: Bootstrap starting.
I20260927 00:55:16.558013 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:55:16.560232 13929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:16.561395 13929 log.cc:824] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:16.565155 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170: No bootstrap required, opened a new log
I20260927 00:55:16.569372 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.560196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.98.62:39695)
0927 00:55:16.560331 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.560336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.560551 (+ 215us) server_negotiation.cc:407] Connection header received
0927 00:55:16.560736 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.560739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.560787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.560836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.561794 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.562555 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.563754 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.564089 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.564163 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.565931 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.565934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.565937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.565964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.567476 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.567945 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.567949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.567950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.568036 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.568351 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.568355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.568357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.568578 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.568681 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.569070 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.569190 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:55:16.570209 13929 raft_consensus.cc:348] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.570580 13929 raft_consensus.cc:374] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:16.570649 13929 raft_consensus.cc:729] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33958c0df3d4d03b6f15e92e1e20170, State: Initialized, Role: FOLLOWER
I20260927 00:55:16.571101 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.560901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.0.98.61:43443)
0927 00:55:16.561042 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.561046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.561179 (+ 133us) server_negotiation.cc:407] Connection header received
0927 00:55:16.561288 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.561291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.561349 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.561445 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.562388 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.563077 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.564238 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.564548 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.564644 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.566442 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.566447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.566450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.566479 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.568201 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.568773 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.568777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.568779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.568897 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.569200 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.569203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.569205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.569819 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.569963 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.570790 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.570930 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:55:16.571386 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.560327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.0.98.60:38163)
0927 00:55:16.560469 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.560475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.561476 (+ 1001us) server_negotiation.cc:407] Connection header received
0927 00:55:16.561747 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.561751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.561819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.561870 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.562771 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.563587 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.564900 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.565465 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.565545 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.567965 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.567968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.567971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.567995 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.569565 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.570026 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.570030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.570031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.570112 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.570407 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.570410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.570412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.570617 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.570713 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.571065 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.571233 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:55:16.571399 13929 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.572139 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.572333 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:55:16.573290 13929 sys_catalog.cc:564] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:55:16.579520 13953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:55:16.579613 13953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:55:16.577370 13860 master_runner.cc:186] Error getting master registration for 127.0.98.62:39695: OK, instance_id { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" instance_seqno: 1790470515950638 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:55:16.580708 13860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:55:16.587888 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 3
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:55:16.598196 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.588557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.0.98.60:38163)
0927 00:55:16.588665 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.588669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.588684 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:55:16.588866 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.588868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.588907 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.588975 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.589555 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.590045 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.591226 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.591516 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.591622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.595056 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.595059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.595061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.595082 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.596438 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.597005 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.597008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.597009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.597089 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.597404 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.597406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.597407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.597591 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.597668 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.597965 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.598074 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:55:16.598301 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.588441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47942)
0927 00:55:16.588617 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.588638 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.588753 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.589028 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.589032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.589296 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.589444 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.589451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.590144 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.590148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.591055 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.591065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.591649 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.591653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.591810 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.592414 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.592429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.594211 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.596557 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.596561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.596563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.596894 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.597173 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.597176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.597178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.597282 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.597675 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.597678 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.597817 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.597926 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.598098 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:55:16.601274 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: Bootstrap starting.
I20260927 00:55:16.604058 13779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:16.604787 13779 log.cc:824] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:16.606582 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: No bootstrap required, opened a new log
I20260927 00:55:16.609562 13779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.609764 13779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:16.609830 13779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85725052bd5f459b92fa2fd158af009a, State: Initialized, Role: FOLLOWER
I20260927 00:55:16.610250 13779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.610700 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.610834 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:55:16.611547 13779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:55:16.618124 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:55:16.618227 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:55:16.616039 13711 master_runner.cc:186] Error getting master registration for 127.0.98.61:43443: OK, instance_id { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" instance_seqno: 1790470516235554 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:55:16.619127 13711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:55:16.619318 13942 raft_consensus.cc:482] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:16.619484 13942 raft_consensus.cc:504] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.620386 13942 leader_election.cc:288] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85725052bd5f459b92fa2fd158af009a (127.0.98.62:39695), a57c87d10eb745538b6d6b7992fd22fe (127.0.98.61:43443)
I20260927 00:55:16.620971 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e33958c0df3d4d03b6f15e92e1e20170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85725052bd5f459b92fa2fd158af009a" is_pre_election: true
I20260927 00:55:16.621290 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33958c0df3d4d03b6f15e92e1e20170 in term 0.
I20260927 00:55:16.621750 13877 leader_election.cc:302] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85725052bd5f459b92fa2fd158af009a, e33958c0df3d4d03b6f15e92e1e20170; no voters:
I20260927 00:55:16.622022 13942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:55:16.622198 13942 raft_consensus.cc:482] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:55:16.622246 13942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:16.623456 13942 raft_consensus.cc:504] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.623754 13942 leader_election.cc:288] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [CANDIDATE]: Term 1 election: Requested vote from peers 85725052bd5f459b92fa2fd158af009a (127.0.98.62:39695), a57c87d10eb745538b6d6b7992fd22fe (127.0.98.61:43443)
I20260927 00:55:16.624344 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e33958c0df3d4d03b6f15e92e1e20170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85725052bd5f459b92fa2fd158af009a"
I20260927 00:55:16.624465 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:16.625778 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33958c0df3d4d03b6f15e92e1e20170 in term 1.
I20260927 00:55:16.626116 13877 leader_election.cc:302] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85725052bd5f459b92fa2fd158af009a, e33958c0df3d4d03b6f15e92e1e20170; no voters:
I20260927 00:55:16.626334 13942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:16.626559 13942 raft_consensus.cc:686] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [term 1 LEADER]: Becoming Leader. State: Replica: e33958c0df3d4d03b6f15e92e1e20170, State: Running, Role: LEADER
I20260927 00:55:16.626935 13942 consensus_queue.cc:237] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.629983 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e33958c0df3d4d03b6f15e92e1e20170. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.630098 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:16.631726 13971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:16.632764 13971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:16.639031 13753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [term 1 FOLLOWER]: Refusing update from remote peer e33958c0df3d4d03b6f15e92e1e20170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:16.639930 13969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:16.641698 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } attempt: 4
I20260927 00:55:16.647982 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e33958c0df3d4d03b6f15e92e1e20170. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.648121 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 00:55:16 dist-test-slave-9fh2 krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:55:16.651926 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.652056 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:16.653003 13972 mvcc.cc:205] Tried to move back new op lower bound from 7333767236143407104 to 7333767236109295616. Current Snapshot: MvccSnapshot[applied={T|T < 7333767236143407104}]
I20260927 00:55:16.655025 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.655133 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:55:16.656497 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.642110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38163 (local address 127.0.0.1:47944)
0927 00:55:16.643263 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:16.643279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:16.643375 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:16.644659 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:16.644663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:16.644886 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:16.645003 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.645009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.646005 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.646009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.646784 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:16.646795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.648267 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.648272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:16.648374 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.648895 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:16.648909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:16.650371 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:16.652335 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.652338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.652341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.652570 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.652872 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:16.652875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:16.652877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:16.652973 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:16.655952 (+ 2979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:16.655956 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:16.656070 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:55:16.656175 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:55:16.656329 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070}
I20260927 00:55:16.657047 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:16.642282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.98.60:38163)
0927 00:55:16.642425 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:16.642430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:16.643331 (+ 901us) server_negotiation.cc:407] Connection header received
0927 00:55:16.643470 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:16.643473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:16.643520 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:16.643581 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:16.645128 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.645908 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.646934 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:16.647284 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:16.647386 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:16.650513 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:16.650517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:16.650520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:16.650546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:16.652181 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.652670 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.652674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.652676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.652771 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:16.653070 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:16.653073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:16.653075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:16.653285 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:16.653394 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:16.656787 (+ 3393us) server_negotiation.cc:299] Negotiation successful
0927 00:55:16.656906 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:55:16.658769 13973 mvcc.cc:205] Tried to move back new op lower bound from 7333767236143407104 to 7333767236109295616. Current Snapshot: MvccSnapshot[applied={T|T < 7333767236143407104}]
I20260927 00:55:16.660302 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: Bootstrap starting.
I20260927 00:55:16.663295 13971 catalog_manager.cc:1347] Generated new cluster ID: fcb3371fc93446c5924b181d57fdf1d9
I20260927 00:55:16.663378 13971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:16.671487 13854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:16.673385 13854 log.cc:824] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:16.677248 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: No bootstrap required, opened a new log
I20260927 00:55:16.684720 13854 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.685309 13854 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:16.685506 13854 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57c87d10eb745538b6d6b7992fd22fe, State: Initialized, Role: FOLLOWER
I20260927 00:55:16.686614 13854 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } }
I20260927 00:55:16.689348 13828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e33958c0df3d4d03b6f15e92e1e20170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57c87d10eb745538b6d6b7992fd22fe" is_pre_election: true
I20260927 00:55:16.689657 13828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33958c0df3d4d03b6f15e92e1e20170 in term 0.
I20260927 00:55:16.689906 13827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e33958c0df3d4d03b6f15e92e1e20170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57c87d10eb745538b6d6b7992fd22fe"
I20260927 00:55:16.690006 13827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:16.691289 13827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33958c0df3d4d03b6f15e92e1e20170 in term 1.
I20260927 00:55:16.692420 13826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [term 1 FOLLOWER]: Refusing update from remote peer e33958c0df3d4d03b6f15e92e1e20170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:16.693349 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.693583 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:55:16.693519 13854 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:16.693658 13942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:16.699455 13971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:55:16.700702 13971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:16.702020 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e33958c0df3d4d03b6f15e92e1e20170. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.702248 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:55:16.702387 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.702504 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:55:16.702579 13989 mvcc.cc:205] Tried to move back new op lower bound from 7333767236143407104 to 7333767236109295616. Current Snapshot: MvccSnapshot[applied={T|T < 7333767236143407104}]
I20260927 00:55:16.703684 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e33958c0df3d4d03b6f15e92e1e20170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85725052bd5f459b92fa2fd158af009a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } peers { permanent_uuid: "a57c87d10eb745538b6d6b7992fd22fe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43443 } } peers { permanent_uuid: "e33958c0df3d4d03b6f15e92e1e20170" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38163 } } }
I20260927 00:55:16.703835 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170 [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:16.710071 13971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e33958c0df3d4d03b6f15e92e1e20170: Generated new TSK 0
I20260927 00:55:16.711076 13971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:55:16.715447 13993 catalog_manager.cc:1259] Loaded cluster ID: fcb3371fc93446c5924b181d57fdf1d9
I20260927 00:55:16.715569 13993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: loading cluster ID for follower catalog manager: success
I20260927 00:55:16.723521 13993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: acquiring CA information for follower catalog manager: success
I20260927 00:55:16.725663 13993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a57c87d10eb745538b6d6b7992fd22fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:55:17.619969 13967 catalog_manager.cc:1259] Loaded cluster ID: fcb3371fc93446c5924b181d57fdf1d9
I20260927 00:55:17.620102 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: loading cluster ID for follower catalog manager: success
I20260927 00:55:17.622095 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: acquiring CA information for follower catalog manager: success
I20260927 00:55:17.623620 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85725052bd5f459b92fa2fd158af009a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:55:17.784600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13711
I20260927 00:55:17.790591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13786
I20260927 00:55:17.797518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 13860
2026-09-27T00:55:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3139 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:55:17 dist-test-slave-9fh2 krb5kdc[14001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:55:17 dist-test-slave-9fh2 krb5kdc[14001](info): set up 2 sockets
Sep 27 00:55:17 dist-test-slave-9fh2 krb5kdc[14001](info): commencing operation
krb5kdc: starting...
W20260927 00:55:18.759927 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:55:18 dist-test-slave-9fh2 krb5kdc[14001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:55:18.895529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:19.005985 14014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:19.006274 14014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:19.006315 14014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:19.010397 14014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:19.010485 14014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:19.010504 14014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:19.010520 14014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:19.010536 14014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:19.016114 14014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:19.017170 14014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:19.018178 14014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:19.027290 14022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:19.027343 14020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:19.027485 14014 server_base.cc:1034] running on GCE node
W20260927 00:55:19.027379 14019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:19.028404 14014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:19.029506 14014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:19.030689 14014 hybrid_clock.cc:648] HybridClock initialized: now 1790470519030663 us; error 50 us; skew 500 ppm
Sep 27 00:55:19 dist-test-slave-9fh2 krb5kdc[14001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:19.033742 14014 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:19.034711 14014 webserver.cc:460] Webserver started at http://127.0.98.62:38749/ using document root <none> and password file <none>
I20260927 00:55:19.035295 14014 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:19.035360 14014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:19.035569 14014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:19.037732 14014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "b22c70b10bd448baa2bc87a321e257af"
format_stamp: "Formatted at 2026-09-27 00:55:19 on dist-test-slave-9fh2"
server_key: "53688b2f77da096918e942a430a58a95"
server_key_iv: "3b06f9a13a2d7c5dcf3c45aa3c2d0210"
server_key_version: "encryptionkey@0"
I20260927 00:55:19.038293 14014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "b22c70b10bd448baa2bc87a321e257af"
format_stamp: "Formatted at 2026-09-27 00:55:19 on dist-test-slave-9fh2"
server_key: "53688b2f77da096918e942a430a58a95"
server_key_iv: "3b06f9a13a2d7c5dcf3c45aa3c2d0210"
server_key_version: "encryptionkey@0"
I20260927 00:55:19.042359 14014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:55:19.045215 14029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:19.046877 14014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:55:19.047022 14014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "b22c70b10bd448baa2bc87a321e257af"
format_stamp: "Formatted at 2026-09-27 00:55:19 on dist-test-slave-9fh2"
server_key: "53688b2f77da096918e942a430a58a95"
server_key_iv: "3b06f9a13a2d7c5dcf3c45aa3c2d0210"
server_key_version: "encryptionkey@0"
I20260927 00:55:19.047206 14014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:19.074165 14014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:19.077406 14014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:19.077574 14014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:19.084903 14014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40897
I20260927 00:55:19.084971 14081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40897 every 8 connection(s)
I20260927 00:55:19.085901 14014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:19.089649 14082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:19.093910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14014
I20260927 00:55:19.094076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:55:19.094424 392 external_mini_cluster.cc:1428] Setting key 7942a1055df0234332c3688e1a8fa0bf
I20260927 00:55:19.097548 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af: Bootstrap starting.
I20260927 00:55:19.100874 14082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:19.102113 14082 log.cc:824] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:55:19 dist-test-slave-9fh2 krb5kdc[14001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:19.105147 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af: No bootstrap required, opened a new log
I20260927 00:55:19.109249 14082 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22c70b10bd448baa2bc87a321e257af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40897 } }
I20260927 00:55:19.109575 14082 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:19.109615 14082 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22c70b10bd448baa2bc87a321e257af, State: Initialized, Role: FOLLOWER
I20260927 00:55:19.110152 14082 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22c70b10bd448baa2bc87a321e257af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40897 } }
I20260927 00:55:19.110306 14082 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:19.110374 14082 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:19.110483 14082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:19.111572 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:19.096219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.0.98.62:40897)
0927 00:55:19.096699 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:19.096716 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:19.096756 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:55:19.097600 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:19.097620 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:19.098054 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:19.098387 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:19.099558 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:19.100637 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:19.101943 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:19.102511 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:19.102681 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:19.105764 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:19.105770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:19.105787 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:19.105822 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:19.108076 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:19.108707 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:19.108711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:19.108716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:19.108841 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:19.109131 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:19.109133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:19.109134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:19.109582 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:19.109809 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:19.110194 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 00:55:19.110703 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260927 00:55:19.111743 14082 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22c70b10bd448baa2bc87a321e257af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40897 } }
I20260927 00:55:19.112162 14082 leader_election.cc:302] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22c70b10bd448baa2bc87a321e257af; no voters:
I20260927 00:55:19.112571 14082 leader_election.cc:288] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:55:19.112749 14087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:19.113049 14087 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [term 1 LEADER]: Becoming Leader. State: Replica: b22c70b10bd448baa2bc87a321e257af, State: Running, Role: LEADER
I20260927 00:55:19.113404 14087 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22c70b10bd448baa2bc87a321e257af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40897 } }
I20260927 00:55:19.114099 14082 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:19.114737 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22c70b10bd448baa2bc87a321e257af. Latest consensus state: current_term: 1 leader_uuid: "b22c70b10bd448baa2bc87a321e257af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22c70b10bd448baa2bc87a321e257af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40897 } } }
I20260927 00:55:19.114868 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:19.115543 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:55:19.115523 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22c70b10bd448baa2bc87a321e257af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22c70b10bd448baa2bc87a321e257af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40897 } } }
I20260927 00:55:19.115617 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:19.115996 14092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:19.119393 14092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:19.126341 14092 catalog_manager.cc:1347] Generated new cluster ID: 1b4c5512a0de4c96b2b585f79b2cbd2c
I20260927 00:55:19.126430 14092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:19.138818 14092 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:55:19.139851 14092 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:19.146178 14092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22c70b10bd448baa2bc87a321e257af: Generated new TSK 0
I20260927 00:55:19.146900 14092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:55:19.172374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14014
2026-09-27T00:55:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1373 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:55:19 dist-test-slave-9fh2 krb5kdc[14109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:55:19 dist-test-slave-9fh2 krb5kdc[14109](info): set up 2 sockets
Sep 27 00:55:19 dist-test-slave-9fh2 krb5kdc[14109](info): commencing operation
krb5kdc: starting...
W20260927 00:55:20.143646 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:55:20 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:55:20.280900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:20.393720 14122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:20.394027 14122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:20.394088 14122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:20.398277 14122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:20.398396 14122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:20.398424 14122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:20.398445 14122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:20.398465 14122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:20.404001 14122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:20.405104 14122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:20.406092 14122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:20.415098 14127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:20.415098 14130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:20.415417 14122 server_base.cc:1034] running on GCE node
W20260927 00:55:20.415186 14128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:20.416180 14122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:20.417313 14122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:20.418493 14122 hybrid_clock.cc:648] HybridClock initialized: now 1790470520418436 us; error 77 us; skew 500 ppm
Sep 27 00:55:20 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:20.421429 14122 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:20.422713 14122 webserver.cc:460] Webserver started at http://127.0.98.62:35825/ using document root <none> and password file <none>
I20260927 00:55:20.423503 14122 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:20.423596 14122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:20.423838 14122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:20.426527 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "587fa53c18a54f7a92a21b2ee54c5a7e"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "d97c53c9674bf63588a1914b32ee0bbe"
server_key_iv: "1b7d1d52fbf03aff157010778dd3212a"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.427311 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "587fa53c18a54f7a92a21b2ee54c5a7e"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "d97c53c9674bf63588a1914b32ee0bbe"
server_key_iv: "1b7d1d52fbf03aff157010778dd3212a"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.432528 14122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:55:20.435842 14137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:20.437709 14122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:55:20.437909 14122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "587fa53c18a54f7a92a21b2ee54c5a7e"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "d97c53c9674bf63588a1914b32ee0bbe"
server_key_iv: "1b7d1d52fbf03aff157010778dd3212a"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.438074 14122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:20.457418 14122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:20.461144 14122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:20.461361 14122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:20.469697 14122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41347
I20260927 00:55:20.469990 14189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41347 every 8 connection(s)
I20260927 00:55:20.470893 14122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:20.474044 14190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:20.479393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14122
I20260927 00:55:20.479519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:55:20.479820 14190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap starting.
I20260927 00:55:20.479903 392 external_mini_cluster.cc:1428] Setting key f35679e34d61dc1fa28bbb6118c42194
I20260927 00:55:20.482463 14190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:20.483206 14190 log.cc:824] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:20.485348 14190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: No bootstrap required, opened a new log
I20260927 00:55:20.489593 14190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:20.489889 14190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:20.489948 14190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587fa53c18a54f7a92a21b2ee54c5a7e, State: Initialized, Role: FOLLOWER
Sep 27 00:55:20 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:20.490383 14190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:20.490512 14190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:20.490578 14190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:20.490672 14190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:20.491631 14190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:20.491971 14190 leader_election.cc:302] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587fa53c18a54f7a92a21b2ee54c5a7e; no voters:
I20260927 00:55:20.492302 14190 leader_election.cc:288] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:55:20.492473 14195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:20.492766 14195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 LEADER]: Becoming Leader. State: Replica: 587fa53c18a54f7a92a21b2ee54c5a7e, State: Running, Role: LEADER
I20260927 00:55:20.493180 14195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:20.493780 14190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:20.495196 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260927 00:55:20.495350 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:20.495683 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587fa53c18a54f7a92a21b2ee54c5a7e. Latest consensus state: current_term: 1 leader_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260927 00:55:20.495796 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:20.497959 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:20.481616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.98.62:41347)
0927 00:55:20.482118 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:20.482138 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:20.482222 (+ 84us) server_negotiation.cc:407] Connection header received
0927 00:55:20.483107 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:20.483135 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:20.483632 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:20.484031 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:20.484976 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:20.486145 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:20.487404 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:20.487973 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:20.488109 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:20.491222 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:20.491229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:20.491254 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:20.491294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:20.493510 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:20.494124 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:20.494130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:20.494138 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:20.494294 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:20.494763 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:20.494768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:20.494770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:20.495492 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:20.496673 (+ 1181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:20.496876 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:55:20.497190 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":145,"threads_started":1}
W20260927 00:55:20.499745 14210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:55:20.499840 14210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:55:20.499948 14206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:20.500887 14206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:20.507802 14206 catalog_manager.cc:1347] Generated new cluster ID: 4f9e0367d07f463c98f1caad8db61906
I20260927 00:55:20.507887 14206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:20.554605 14206 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:55:20.555677 14206 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:20.566429 14206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Generated new TSK 0
I20260927 00:55:20.567674 14206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:55:20.602896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:20.713721 14216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:20.714006 14216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:20.714051 14216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:20.718159 14216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:20.718241 14216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:20.718303 14216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:55:20.723721 14216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:20.724783 14216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:20.725754 14216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:20.735526 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:20.735544 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:20.735634 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:20.735935 14216 server_base.cc:1034] running on GCE node
I20260927 00:55:20.736342 14216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:20.736966 14216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:20.738137 14216 hybrid_clock.cc:648] HybridClock initialized: now 1790470520738115 us; error 59 us; skew 500 ppm
Sep 27 00:55:20 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:20.741338 14216 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:55:20.742305 14216 webserver.cc:460] Webserver started at http://127.0.98.1:39237/ using document root <none> and password file <none>
I20260927 00:55:20.742942 14216 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:20.743010 14216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:20.743255 14216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:20.745354 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "a22cab90894bf9ed511d8c34b03d2815"
server_key_iv: "1a0d6af1b253589893ad8e675f0160a5"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.745898 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "a22cab90894bf9ed511d8c34b03d2815"
server_key_iv: "1a0d6af1b253589893ad8e675f0160a5"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.750020 14216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:55:20.752785 14231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:20.754419 14216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:55:20.754557 14216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "a22cab90894bf9ed511d8c34b03d2815"
server_key_iv: "1a0d6af1b253589893ad8e675f0160a5"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.754668 14216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:20.767915 14216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:20.771395 14216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:20.771585 14216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:20.772178 14216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:20.773043 14216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:20.773092 14216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:20.773144 14216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:20.773172 14216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:20.782814 14216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42213
I20260927 00:55:20.782893 14344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42213 every 8 connection(s)
I20260927 00:55:20.783792 14216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:55:20.789873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14216
I20260927 00:55:20.789987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:55:20.790263 392 external_mini_cluster.cc:1428] Setting key 880681baa361d3c77b37a61e9a17023f
Sep 27 00:55:20 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:20.799978 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:20.786193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40631 (local address 127.0.98.62:41347)
0927 00:55:20.786400 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:20.786404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:20.787601 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 00:55:20.788798 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:20.788801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:20.788872 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:20.788954 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:20.790514 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:20.791236 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:20.792514 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:20.792790 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:20.792887 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:20.796391 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:20.796395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:20.796397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:20.796427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:20.797778 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:20.798364 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:20.798367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:20.798368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:20.798453 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:20.798787 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:20.798793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:20.798795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:20.798995 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:20.799079 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:20.799660 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 00:55:20.799806 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:55:20.800899 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:20.786558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:40631)
0927 00:55:20.787397 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:20.787456 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:20.788471 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:20.789205 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:20.789214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:20.789676 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:20.790347 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:20.790370 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:20.791363 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:20.791367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:20.792387 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:20.792398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:20.792957 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:20.792961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:20.793173 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:20.794043 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:20.794066 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:20.796176 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:20.797911 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:20.797919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:20.797944 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:20.798264 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:20.798536 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:20.798540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:20.798542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:20.798689 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:20.799095 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:20.799102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:20.799426 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 00:55:20.799606 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:55:20.799995 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":112,"threads_started":1}
I20260927 00:55:20.802595 14345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260927 00:55:20.802907 14345 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:55:20.803608 14345 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260927 00:55:20.805877 14154 ts_manager.cc:194] Registered new tserver with Master: 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213)
I20260927 00:55:20.807293 14154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:55:20.822782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:20.954308 14350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:20.954689 14350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:20.954741 14350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:20.959286 14350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:20.959414 14350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:20.959496 14350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:55:20.965904 14350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:20.967408 14350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:20.968660 14350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:20.980937 14356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:20.980904 14358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:20.980912 14355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:20.981357 14350 server_base.cc:1034] running on GCE node
I20260927 00:55:20.981868 14350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:20.982595 14350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:20.983771 14350 hybrid_clock.cc:648] HybridClock initialized: now 1790470520983745 us; error 50 us; skew 500 ppm
Sep 27 00:55:20 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:20.986744 14350 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:55:20.987790 14350 webserver.cc:460] Webserver started at http://127.0.98.2:38503/ using document root <none> and password file <none>
I20260927 00:55:20.988394 14350 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:20.988461 14350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:20.988667 14350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:20.990839 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "98d074e65621433a8bad3480c13078ab"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "ef74ab39132898ba66f810ad6495975a"
server_key_iv: "90562d29e5407ea475be12b001488ffb"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.991454 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "98d074e65621433a8bad3480c13078ab"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "ef74ab39132898ba66f810ad6495975a"
server_key_iv: "90562d29e5407ea475be12b001488ffb"
server_key_version: "encryptionkey@0"
I20260927 00:55:20.996135 14350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:55:20.999230 14365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.000903 14350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:55:21.001070 14350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "98d074e65621433a8bad3480c13078ab"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "ef74ab39132898ba66f810ad6495975a"
server_key_iv: "90562d29e5407ea475be12b001488ffb"
server_key_version: "encryptionkey@0"
I20260927 00:55:21.001189 14350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:21.022082 14350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:21.025465 14350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:21.025955 14350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:21.026651 14350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:21.027622 14350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:21.027681 14350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.027737 14350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:21.027765 14350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.038086 14350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45269
I20260927 00:55:21.038147 14478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45269 every 8 connection(s)
I20260927 00:55:21.039099 14350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:55:21.040932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14350
I20260927 00:55:21.041100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:55:21.041374 392 external_mini_cluster.cc:1428] Setting key c55e81133902b2904cd23a874ebfbd70
Sep 27 00:55:21 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:55:21.056097 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.041639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36973 (local address 127.0.98.62:41347)
0927 00:55:21.041793 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.041797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.042806 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 00:55:21.044607 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.044611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.044670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.044770 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:21.046190 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.047003 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.048410 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.048774 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.048930 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.052033 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:21.052037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:21.052049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:21.052088 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:21.053757 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.054348 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.054351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.054353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.054444 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.054811 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.054813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.054814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.055041 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:21.055132 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.055788 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.055937 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:55:21.057145 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.042106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:36973)
0927 00:55:21.042634 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.042696 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.044263 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.045000 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.045010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.045463 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:21.045991 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.046005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.047221 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.047224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.048257 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.048269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.048990 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.048994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.049173 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.049918 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:21.049936 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:21.051800 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:21.053901 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:21.053907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:21.053922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:21.054209 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:21.054549 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:21.054551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:21.054553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:21.054705 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:21.055182 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:21.055189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:21.055512 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.055699 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.056079 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260927 00:55:21.058599 14479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260927 00:55:21.058959 14479 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:21.059739 14479 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260927 00:55:21.061307 14154 ts_manager.cc:194] Registered new tserver with Master: 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
I20260927 00:55:21.062019 14154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:55:21.074357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:21.192394 14484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:21.192687 14484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:21.192730 14484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:21.197096 14484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:21.197190 14484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:21.197254 14484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:55:21.202872 14484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:21.204097 14484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:21.205192 14484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:21.216102 14490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:21.216118 14489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:21.216148 14492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:21.216476 14484 server_base.cc:1034] running on GCE node
I20260927 00:55:21.217020 14484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:21.217700 14484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:21.218887 14484 hybrid_clock.cc:648] HybridClock initialized: now 1790470521218859 us; error 49 us; skew 500 ppm
Sep 27 00:55:21 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:21.225431 14484 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:55:21.226500 14484 webserver.cc:460] Webserver started at http://127.0.98.3:40247/ using document root <none> and password file <none>
I20260927 00:55:21.227064 14484 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:21.227130 14484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:21.227360 14484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:21.229404 14484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
format_stamp: "Formatted at 2026-09-27 00:55:21 on dist-test-slave-9fh2"
server_key: "a6697252e45d6287254796a0c75062a8"
server_key_iv: "3671d99d163e175695c7b85f095339e1"
server_key_version: "encryptionkey@0"
I20260927 00:55:21.229956 14484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
format_stamp: "Formatted at 2026-09-27 00:55:21 on dist-test-slave-9fh2"
server_key: "a6697252e45d6287254796a0c75062a8"
server_key_iv: "3671d99d163e175695c7b85f095339e1"
server_key_version: "encryptionkey@0"
I20260927 00:55:21.234108 14484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 00:55:21.237277 14499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.238855 14484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:55:21.239003 14484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
format_stamp: "Formatted at 2026-09-27 00:55:21 on dist-test-slave-9fh2"
server_key: "a6697252e45d6287254796a0c75062a8"
server_key_iv: "3671d99d163e175695c7b85f095339e1"
server_key_version: "encryptionkey@0"
I20260927 00:55:21.239132 14484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:21.270131 14484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:21.273851 14484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:21.274050 14484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:21.274664 14484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:21.275549 14484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:21.275599 14484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.275650 14484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:21.275681 14484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.286113 14484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36901
I20260927 00:55:21.286175 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36901 every 8 connection(s)
I20260927 00:55:21.287110 14484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:55:21.293111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14484
I20260927 00:55:21.293227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:55:21.293483 392 external_mini_cluster.cc:1428] Setting key 8c435878ce7748ad0f6dbc8aed7a4882
Sep 27 00:55:21 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:21.303246 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.289543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46673 (local address 127.0.98.62:41347)
0927 00:55:21.289708 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.289712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.290819 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 00:55:21.292560 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.292563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.292629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.292742 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:21.294572 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.295287 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.296717 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.296942 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.297032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.299872 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:21.299875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:21.299876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:21.299903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:21.301220 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.301730 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.301734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.301736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.301807 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.302123 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.302126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.302128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.302323 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:21.302409 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.302909 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.303034 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:55:21.304070 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.289993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:46673)
0927 00:55:21.290614 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.290677 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.292240 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.293045 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.293055 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.293606 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:21.294391 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.294414 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.295460 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.295466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.296579 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.296597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.297093 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.297098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.297280 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.297919 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:21.297954 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:21.299690 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:21.301343 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:21.301350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:21.301367 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:21.301634 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:21.301888 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:21.301891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:21.301893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:21.302037 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:21.302418 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:21.302425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:21.302683 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.302855 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.303251 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":171,"threads_started":1}
I20260927 00:55:21.305462 14613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260927 00:55:21.305879 14613 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:21.306702 14613 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260927 00:55:21.307952 14154 ts_manager.cc:194] Registered new tserver with Master: e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901)
I20260927 00:55:21.308768 14154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46673
I20260927 00:55:21.320622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:55:21.330472 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.322964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.0.98.62:41347)
0927 00:55:21.323131 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.323136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.323400 (+ 264us) server_negotiation.cc:407] Connection header received
0927 00:55:21.323541 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.323544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.323588 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.323654 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:21.324357 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.324879 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.325673 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.325889 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.325973 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.327265 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:21.327269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:21.327272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:21.327296 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:21.328819 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.329247 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.329251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.329252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.329328 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.329629 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.329632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.329634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.329861 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:21.329977 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.330188 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.330302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:55:21.334650 14154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:55:21.338052 14154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:55:21.354692 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.348699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.0.98.2:45269)
0927 00:55:21.349541 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.349554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.349576 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:55:21.349652 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.349657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.349888 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.350430 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:21.350596 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.351570 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.353023 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.353870 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.353981 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.354140 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.354416 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.354523 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":144,"threads_started":1}
I20260927 00:55:21.354753 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.348500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.0.1:35740)
0927 00:55:21.348981 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.349060 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.349227 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.350074 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.350081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.350104 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:21.350267 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.350274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.351684 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.351688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.352880 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.352890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.354003 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.354006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.354158 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.354216 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.354369 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.354554 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":30,"thread_start_us":134,"threads_started":1}
I20260927 00:55:21.355226 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.349437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.0.1:50658)
0927 00:55:21.350346 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.350360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.350441 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.350948 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.350951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.350965 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:21.351044 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.351049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.352418 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.352422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.353501 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.353512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.354764 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.354768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.354872 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.354898 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.354969 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.355058 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260927 00:55:21.355618 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.349280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42213 (local address 127.0.0.1:55254)
0927 00:55:21.350088 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.350100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.350168 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.350704 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.350707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.350724 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:21.350806 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.350812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.352608 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.352611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.353855 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.353867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.355132 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.355136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.355267 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.355297 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.355372 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.355467 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":20,"thread_start_us":112,"threads_started":1}
I20260927 00:55:21.355942 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.349944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.98.3:36901)
0927 00:55:21.350362 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.350372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.350422 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:55:21.350571 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.350577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.350767 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.350910 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:21.351184 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.352288 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.353638 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.354430 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.354527 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.354687 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.355409 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.355548 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":114,"threads_started":1}
I20260927 00:55:21.356170 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.349510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.0.98.1:42213)
0927 00:55:21.350162 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.350170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.350199 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:55:21.350312 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.350318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.350549 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.350726 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:21.350892 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.352201 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.354062 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.354827 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.354938 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.355073 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.355920 (+ 847us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.356031 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":106,"threads_started":1}
I20260927 00:55:21.358425 14413 tablet_service.cc:1467] Processing CreateTablet for tablet 023705c280434a099169636d77ea24c8 (DEFAULT_TABLE table=test-table [id=7fc56d0133ce40be91607553c82e8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:21.359472 14413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023705c280434a099169636d77ea24c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:21.366110 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 023705c280434a099169636d77ea24c8 (DEFAULT_TABLE table=test-table [id=7fc56d0133ce40be91607553c82e8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:21.366632 14547 tablet_service.cc:1467] Processing CreateTablet for tablet 023705c280434a099169636d77ea24c8 (DEFAULT_TABLE table=test-table [id=7fc56d0133ce40be91607553c82e8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:21.367192 14630 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Bootstrap starting.
I20260927 00:55:21.367486 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023705c280434a099169636d77ea24c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:21.368005 14547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023705c280434a099169636d77ea24c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:21.370669 14630 tablet_bootstrap.cc:654] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:21.371905 14630 log.cc:824] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:21.374143 14632 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap starting.
I20260927 00:55:21.374644 14630 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: No bootstrap required, opened a new log
I20260927 00:55:21.374869 14630 ts_tablet_manager.cc:1397] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:55:21.377250 14633 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Bootstrap starting.
I20260927 00:55:21.377908 14632 tablet_bootstrap.cc:654] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:21.378471 14630 raft_consensus.cc:348] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.378811 14630 raft_consensus.cc:374] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:21.378870 14630 raft_consensus.cc:729] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d074e65621433a8bad3480c13078ab, State: Initialized, Role: FOLLOWER
I20260927 00:55:21.378945 14632 log.cc:824] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:21.379354 14630 consensus_queue.cc:260] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.379948 14633 tablet_bootstrap.cc:654] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:21.380493 14630 ts_tablet_manager.cc:1428] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:55:21.380983 14479 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260927 00:55:21.381246 14633 log.cc:824] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:21.381768 14632 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: No bootstrap required, opened a new log
I20260927 00:55:21.382037 14632 ts_tablet_manager.cc:1397] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:55:21.383189 14633 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: No bootstrap required, opened a new log
I20260927 00:55:21.383386 14633 ts_tablet_manager.cc:1397] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:55:21.386722 14633 raft_consensus.cc:348] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.387032 14633 raft_consensus.cc:374] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:21.387096 14633 raft_consensus.cc:729] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d484a69ee7428fa60c59e1f4c04a2d, State: Initialized, Role: FOLLOWER
I20260927 00:55:21.386948 14632 raft_consensus.cc:348] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.387298 14632 raft_consensus.cc:374] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:21.387355 14632 raft_consensus.cc:729] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3397a2a25fdd4180a0d5d0cb8ee9a354, State: Initialized, Role: FOLLOWER
I20260927 00:55:21.387611 14633 consensus_queue.cc:260] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.387841 14632 consensus_queue.cc:260] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.388250 14613 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260927 00:55:21.388692 14633 ts_tablet_manager.cc:1428] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:55:21.389009 14632 ts_tablet_manager.cc:1428] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 00:55:21.389338 14345 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260927 00:55:21.449913 14635 raft_consensus.cc:482] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:21.450141 14635 raft_consensus.cc:504] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.451345 14635 leader_election.cc:288] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213)
I20260927 00:55:21.456061 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.451838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41651 (local address 127.0.98.1:42213)
0927 00:55:21.452003 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.452008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.452025 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:55:21.452168 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.452171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.452226 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.452333 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:21.452747 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.453349 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.454638 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.455388 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.455500 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.455593 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.455815 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.455903 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:55:21.456069 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.451682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42213 (local address 127.0.98.2:41651)
0927 00:55:21.451900 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.451916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.452015 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.452383 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.452388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.452421 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:21.452593 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.452602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.453476 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.453480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.454446 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.454458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.455515 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.455519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.455690 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.455711 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.455784 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.455888 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260927 00:55:21.456238 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.452105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45417 (local address 127.0.98.3:36901)
0927 00:55:21.452291 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.452296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.452318 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:55:21.452388 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.452391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.452443 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.452542 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:21.452944 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.453516 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.454975 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.455670 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.455771 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.455854 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.456018 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.456095 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 00:55:21.456262 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.451704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.98.2:45417)
0927 00:55:21.452213 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.452226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.452308 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.452605 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.452608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.452633 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:21.452810 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.452819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.453729 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.453732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.454772 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.454785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.455787 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.455790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.455903 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.455927 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.455993 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.456081 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":1,"thread_start_us":197,"threads_started":1}
I20260927 00:55:21.457015 14567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023705c280434a099169636d77ea24c8" candidate_uuid: "98d074e65621433a8bad3480c13078ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" is_pre_election: true
I20260927 00:55:21.457018 14299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023705c280434a099169636d77ea24c8" candidate_uuid: "98d074e65621433a8bad3480c13078ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" is_pre_election: true
I20260927 00:55:21.457343 14567 raft_consensus.cc:2454] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98d074e65621433a8bad3480c13078ab in term 0.
I20260927 00:55:21.457427 14299 raft_consensus.cc:2454] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98d074e65621433a8bad3480c13078ab in term 0.
I20260927 00:55:21.457942 14366 leader_election.cc:302] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354, 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d; no voters:
I20260927 00:55:21.458267 14635 raft_consensus.cc:2781] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:55:21.458366 14635 raft_consensus.cc:482] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:55:21.458426 14635 raft_consensus.cc:3037] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:21.459560 14635 raft_consensus.cc:504] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.459930 14635 leader_election.cc:288] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [CANDIDATE]: Term 1 election: Requested vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213)
I20260927 00:55:21.460446 14299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023705c280434a099169636d77ea24c8" candidate_uuid: "98d074e65621433a8bad3480c13078ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354"
I20260927 00:55:21.460481 14567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023705c280434a099169636d77ea24c8" candidate_uuid: "98d074e65621433a8bad3480c13078ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
I20260927 00:55:21.460597 14567 raft_consensus.cc:3037] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:21.460608 14299 raft_consensus.cc:3037] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:21.461640 14567 raft_consensus.cc:2454] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d074e65621433a8bad3480c13078ab in term 1.
I20260927 00:55:21.461952 14366 leader_election.cc:302] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d; no voters:
I20260927 00:55:21.462106 14299 raft_consensus.cc:2454] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d074e65621433a8bad3480c13078ab in term 1.
I20260927 00:55:21.462191 14635 raft_consensus.cc:2781] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:21.462437 14635 raft_consensus.cc:686] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 1 LEADER]: Becoming Leader. State: Replica: 98d074e65621433a8bad3480c13078ab, State: Running, Role: LEADER
I20260927 00:55:21.462880 14635 consensus_queue.cc:237] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:21.466995 14153 catalog_manager.cc:5495] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 98d074e65621433a8bad3480c13078ab (127.0.98.2). New cstate: current_term: 1 leader_uuid: "98d074e65621433a8bad3480c13078ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } health_report { overall_health: UNKNOWN } } }
I20260927 00:55:21.483199 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.479796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.0.98.2:45269)
0927 00:55:21.479946 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.479950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.479969 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:55:21.480146 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.480148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.480204 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.480299 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:55:21.480515 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.481020 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.481805 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.482016 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.482099 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.482363 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:55:21.482781 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:55:21.482891 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.482951 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.483006 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:55:21.490406 14567 raft_consensus.cc:1261] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Refusing update from remote peer 98d074e65621433a8bad3480c13078ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:21.490509 14299 raft_consensus.cc:1261] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Refusing update from remote peer 98d074e65621433a8bad3480c13078ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:21.491245 14640 consensus_queue.cc:1035] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:21.491573 14635 consensus_queue.cc:1035] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:21.501132 14645 mvcc.cc:205] Tried to move back new op lower bound from 7333767256015138816 to 7333767255915212800. Current Snapshot: MvccSnapshot[applied={T|T < 7333767256015138816}]
I20260927 00:55:21.508837 14154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:21.509034 14154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:21.511751 14154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Sending DeleteTablet for 3 replicas of tablet 023705c280434a099169636d77ea24c8
I20260927 00:55:21.512961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14216
I20260927 00:55:21.512894 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 023705c280434a099169636d77ea24c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55254
I20260927 00:55:21.512894 14413 tablet_service.cc:1514] Processing DeleteTablet for tablet 023705c280434a099169636d77ea24c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35740
I20260927 00:55:21.514178 14547 tablet_service.cc:1514] Processing DeleteTablet for tablet 023705c280434a099169636d77ea24c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50658
I20260927 00:55:21.514478 14653 tablet_replica.cc:330] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: stopping tablet replica
I20260927 00:55:21.514690 14654 tablet_replica.cc:330] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: stopping tablet replica
I20260927 00:55:21.514824 14653 raft_consensus.cc:2229] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:55:21.515053 14654 raft_consensus.cc:2229] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:21.515292 14653 raft_consensus.cc:2258] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:21.515509 14654 raft_consensus.cc:2258] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:21.516999 14653 ts_tablet_manager.cc:1905] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:21.516999 14654 ts_tablet_manager.cc:1905] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:21.519743 14653 ts_tablet_manager.cc:1918] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:21.519743 14654 ts_tablet_manager.cc:1918] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:21.519861 14653 log.cc:1196] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/023705c280434a099169636d77ea24c8
I20260927 00:55:21.519984 14654 log.cc:1196] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/023705c280434a099169636d77ea24c8
I20260927 00:55:21.520177 14653 ts_tablet_manager.cc:1939] T 023705c280434a099169636d77ea24c8 P 98d074e65621433a8bad3480c13078ab: Deleting consensus metadata
I20260927 00:55:21.520401 14654 ts_tablet_manager.cc:1939] T 023705c280434a099169636d77ea24c8 P e1d484a69ee7428fa60c59e1f4c04a2d: Deleting consensus metadata
I20260927 00:55:21.521652 14138 catalog_manager.cc:4841] TS 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): tablet 023705c280434a099169636d77ea24c8 (table test-table [id=7fc56d0133ce40be91607553c82e8e8b]) successfully deleted
I20260927 00:55:21.521878 14138 catalog_manager.cc:4841] TS e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): tablet 023705c280434a099169636d77ea24c8 (table test-table [id=7fc56d0133ce40be91607553c82e8e8b]) successfully deleted
W20260927 00:55:21.522583 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42213 (error 108)
I20260927 00:55:21.522874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14350
I20260927 00:55:21.525259 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.524896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42213 (local address 127.0.0.1:55266)
0927 00:55:21.525066 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.525151 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:55:21.525452 14141 catalog_manager.cc:4584] TS 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 023705c280434a099169636d77ea24c8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42213: connect: Connection refused (error 111)
I20260927 00:55:21.529556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14484
I20260927 00:55:21.536366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14122
I20260927 00:55:21.544492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--webserver_interface=127.0.98.62
--webserver_port=35825
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:21.655853 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:21.656131 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:21.656172 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:21.660145 14656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:21.660207 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:21.660225 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:21.660240 14656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:21.660254 14656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:21.665504 14656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:21.666507 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:21.667521 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:21.681737 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:21.681735 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:21.682034 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:21.682278 14656 server_base.cc:1034] running on GCE node
I20260927 00:55:21.683122 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:21.684782 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:21.685990 14656 hybrid_clock.cc:648] HybridClock initialized: now 1790470521685950 us; error 63 us; skew 500 ppm
Sep 27 00:55:21 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:21.689532 14656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:21.690856 14656 webserver.cc:460] Webserver started at http://127.0.98.62:35825/ using document root <none> and password file <none>
I20260927 00:55:21.691660 14656 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:21.691751 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:21.697074 14656 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:55:21.700392 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.702198 14656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:55:21.702378 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "587fa53c18a54f7a92a21b2ee54c5a7e"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "d97c53c9674bf63588a1914b32ee0bbe"
server_key_iv: "1b7d1d52fbf03aff157010778dd3212a"
server_key_version: "encryptionkey@0"
I20260927 00:55:21.703342 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:21.722560 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:21.726528 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:21.726763 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:21.736474 14656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41347
I20260927 00:55:21.736789 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41347 every 8 connection(s)
I20260927 00:55:21.738009 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:21.742930 14725 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:55:21.743850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14656
I20260927 00:55:21.744321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41347
--webserver_port=39237
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:55:21.744837 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap starting.
I20260927 00:55:21.762033 14725 log.cc:824] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:21.770864 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:55:21.771284 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap complete.
I20260927 00:55:21.775290 14725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:21.775727 14725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587fa53c18a54f7a92a21b2ee54c5a7e, State: Initialized, Role: FOLLOWER
I20260927 00:55:21.776221 14725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:21.776353 14725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:21.776427 14725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:21.776521 14725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:55:21.778563 14725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:21.778898 14725 leader_election.cc:302] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587fa53c18a54f7a92a21b2ee54c5a7e; no voters:
I20260927 00:55:21.779258 14725 leader_election.cc:288] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:55:21.779525 14729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:55:21.780045 14729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 LEADER]: Becoming Leader. State: Replica: 587fa53c18a54f7a92a21b2ee54c5a7e, State: Running, Role: LEADER
I20260927 00:55:21.780464 14729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:21.780648 14725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:21.782274 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587fa53c18a54f7a92a21b2ee54c5a7e. Latest consensus state: current_term: 2 leader_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260927 00:55:21.782446 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:21.782804 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260927 00:55:21.782905 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:21.785983 14744 catalog_manager.cc:1259] Loaded cluster ID: 4f9e0367d07f463c98f1caad8db61906
I20260927 00:55:21.786068 14744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: loading cluster ID for follower catalog manager: success
W20260927 00:55:21.787083 14744 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260927 00:55:21.787181 14744 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260927 00:55:21.787240 14743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:21.789737 14743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:21.790249 14743 catalog_manager.cc:1259] Loaded cluster ID: 4f9e0367d07f463c98f1caad8db61906
I20260927 00:55:21.790323 14743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:21.791428 14743 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 00:55:21.796244 14743 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 00:55:21.802913 14743 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:21.803887 14743 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Loaded TSK: 0
I20260927 00:55:21.805279 14743 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:55:21.883076 14727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:21.883458 14727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:21.883526 14727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:21.888008 14727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:21.888096 14727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:21.888190 14727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:55:21.894423 14727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39237
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:21.895916 14727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:21.897250 14727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:21.908684 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:21.908857 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:21.908857 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:21.909502 14727 server_base.cc:1034] running on GCE node
I20260927 00:55:21.910046 14727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:21.910887 14727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:21.912107 14727 hybrid_clock.cc:648] HybridClock initialized: now 1790470521912070 us; error 52 us; skew 500 ppm
Sep 27 00:55:21 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:21.915455 14727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:55:21.916761 14727 webserver.cc:460] Webserver started at http://127.0.98.1:39237/ using document root <none> and password file <none>
I20260927 00:55:21.917492 14727 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:21.917593 14727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:21.923094 14727 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:55:21.926128 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:21.928020 14727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:21.928190 14727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "a22cab90894bf9ed511d8c34b03d2815"
server_key_iv: "1a0d6af1b253589893ad8e675f0160a5"
server_key_version: "encryptionkey@0"
I20260927 00:55:21.928826 14727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:21.945380 14727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:21.948611 14727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:21.948782 14727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:21.949342 14727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:21.950680 14769 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:55:21.956107 14727 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:55:21.956223 14727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 00:55:21.956307 14727 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:55:21.958290 14727 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:55:21.958343 14727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:55:21.958891 14769 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap starting.
I20260927 00:55:21.968528 14727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42213
I20260927 00:55:21.968632 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42213 every 8 connection(s)
I20260927 00:55:21.969625 14727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:55:21.972925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14727
I20260927 00:55:21.973488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41347
--webserver_port=38503
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:55:21 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:21.983240 14769 log.cc:824] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:21.989018 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.972083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60613 (local address 127.0.98.62:41347)
0927 00:55:21.972505 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.972520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.973844 (+ 1324us) server_negotiation.cc:407] Connection header received
0927 00:55:21.975056 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:21.975079 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:21.975704 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:21.976326 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:21.977614 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.978676 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.980210 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.980674 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.980833 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.983804 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:21.983809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:21.983824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:21.983858 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:21.985778 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.986276 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.986282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.986286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.986410 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:21.986722 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:21.986725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:21.986726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:21.987109 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:21.987302 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:21.987831 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 00:55:21.988085 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
I20260927 00:55:21.989612 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.972196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:60613)
0927 00:55:21.972921 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.972988 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.974311 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:21.976399 (+ 2088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:21.976408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:21.976793 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:21.977367 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.977384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.978882 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.978886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.980082 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:21.980094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:21.980852 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:21.980856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:21.981079 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:21.981712 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:21.981729 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:21.983606 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:21.985898 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:21.985904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:21.985918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:21.986164 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:21.986500 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:21.986503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:21.986505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:21.986635 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:21.987281 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:21.987288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:21.987537 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 00:55:21.987706 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:55:21.988081 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":205,"threads_started":1}
I20260927 00:55:21.991899 14877 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260927 00:55:21.992136 14877 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:21.992676 14877 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260927 00:55:21.994817 14769 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:55:21.995131 14689 ts_manager.cc:194] Registered new tserver with Master: 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213)
I20260927 00:55:21.995364 14769 tablet_bootstrap.cc:492] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap complete.
I20260927 00:55:21.996830 14769 ts_tablet_manager.cc:1397] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.025s
I20260927 00:55:21.998710 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60613
I20260927 00:55:22.000958 14769 raft_consensus.cc:348] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.001610 14769 raft_consensus.cc:729] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3397a2a25fdd4180a0d5d0cb8ee9a354, State: Initialized, Role: FOLLOWER
I20260927 00:55:22.002213 14769 consensus_queue.cc:260] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.002966 14877 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260927 00:55:22.003482 14769 ts_tablet_manager.cc:1428] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260927 00:55:22.005352 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.998076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42213 (local address 127.0.0.1:55268)
0927 00:55:21.999669 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:21.999715 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:21.999865 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.000888 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.000895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.000919 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:22.001039 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.001046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.002204 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.002207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.003379 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.003390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.004749 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.004753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.004867 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.004908 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.005039 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.005182 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"thread_start_us":125,"threads_started":1}
I20260927 00:55:22.005470 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:21.998905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.0.98.1:42213)
0927 00:55:21.999365 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:21.999378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:21.999997 (+ 619us) server_negotiation.cc:407] Connection header received
0927 00:55:22.000064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.000069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.000590 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.000737 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:22.001167 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.002100 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.003691 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.004436 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.004520 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.004644 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.005130 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.005300 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":148,"threads_started":1}
I20260927 00:55:22.007246 14811 tablet_service.cc:1514] Processing DeleteTablet for tablet 023705c280434a099169636d77ea24c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55268
I20260927 00:55:22.008404 14886 tablet_replica.cc:330] stopping tablet replica
I20260927 00:55:22.008828 14886 raft_consensus.cc:2229] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:22.009130 14886 raft_consensus.cc:2258] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:22.010612 14886 ts_tablet_manager.cc:1905] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:22.016986 14886 ts_tablet_manager.cc:1918] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:22.017114 14886 log.cc:1196] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/023705c280434a099169636d77ea24c8
I20260927 00:55:22.017416 14886 ts_tablet_manager.cc:1939] T 023705c280434a099169636d77ea24c8 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Deleting consensus metadata
I20260927 00:55:22.019886 14676 catalog_manager.cc:4841] TS 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213): tablet 023705c280434a099169636d77ea24c8 (table test-table [id=7fc56d0133ce40be91607553c82e8e8b]) successfully deleted
W20260927 00:55:22.122877 14881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:22.123306 14881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:22.123365 14881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:22.127856 14881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:22.127936 14881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:22.128002 14881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:55:22.134276 14881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38503
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:22.135833 14881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:22.137117 14881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:22.149411 14892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:22.149499 14891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:22.149417 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:22.149533 14881 server_base.cc:1034] running on GCE node
I20260927 00:55:22.150198 14881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:22.150947 14881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:22.152138 14881 hybrid_clock.cc:648] HybridClock initialized: now 1790470522152112 us; error 51 us; skew 500 ppm
Sep 27 00:55:22 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:22.156061 14881 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:55:22.157328 14881 webserver.cc:460] Webserver started at http://127.0.98.2:38503/ using document root <none> and password file <none>
I20260927 00:55:22.158051 14881 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:22.158144 14881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:22.163399 14881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:22.166435 14901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.168414 14881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:55:22.168612 14881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "98d074e65621433a8bad3480c13078ab"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "ef74ab39132898ba66f810ad6495975a"
server_key_iv: "90562d29e5407ea475be12b001488ffb"
server_key_version: "encryptionkey@0"
I20260927 00:55:22.169366 14881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:22.184020 14881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:22.188221 14881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:22.188464 14881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:22.189244 14881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:22.190357 14881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:22.190409 14881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.190470 14881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:22.190500 14881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.201967 14881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45269
I20260927 00:55:22.202015 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45269 every 8 connection(s)
I20260927 00:55:22.203325 14881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:55:22.203768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 14881
I20260927 00:55:22.204241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36901
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41347
--webserver_port=40247
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:55:22 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:22.222873 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.206049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33769 (local address 127.0.98.62:41347)
0927 00:55:22.206251 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.206257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.207423 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 00:55:22.209338 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.209342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.209422 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.209629 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:22.211542 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.212370 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.214264 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.214599 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.214762 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.218186 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:22.218191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:22.218193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:22.218242 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:22.220121 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:22.220860 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:22.220863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:22.220865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:22.220964 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:22.221404 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:22.221408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:22.221409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:22.221732 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:22.221844 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.222548 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.222712 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:55:22.223912 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.206573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:33769)
0927 00:55:22.207222 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.207295 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:22.208994 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.209875 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.209886 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.210465 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:22.211324 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.211349 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.212540 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.212545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.214081 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.214096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.214853 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.214858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.215084 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.215950 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:22.215972 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:22.217917 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:22.220271 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:22.220278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:22.220297 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:22.220723 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:22.221052 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:22.221055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:22.221057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:22.221233 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:22.221862 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:22.221871 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:22.222225 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.222437 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.222797 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":140,"threads_started":1}
I20260927 00:55:22.225474 15015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260927 00:55:22.225934 15015 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:22.226647 15015 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260927 00:55:22.228047 14689 ts_manager.cc:194] Registered new tserver with Master: 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
I20260927 00:55:22.228724 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33769
W20260927 00:55:22.330192 15017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:22.330480 15017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:22.330523 15017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:22.335719 15017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:22.335831 15017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:22.335912 15017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:55:22.342996 15017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40247
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:22.344589 15017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:22.346128 15017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:22.358649 15023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:22.358768 15017 server_base.cc:1034] running on GCE node
W20260927 00:55:22.358649 15026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:22.358639 15024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:22.359694 15017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:22.360569 15017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:22.361801 15017 hybrid_clock.cc:648] HybridClock initialized: now 1790470522361764 us; error 58 us; skew 500 ppm
Sep 27 00:55:22 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:22.365263 15017 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:55:22.366490 15017 webserver.cc:460] Webserver started at http://127.0.98.3:40247/ using document root <none> and password file <none>
I20260927 00:55:22.367311 15017 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:22.367417 15017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:22.372537 15017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:55:22.375438 15033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.377091 15017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:55:22.377246 15017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
format_stamp: "Formatted at 2026-09-27 00:55:21 on dist-test-slave-9fh2"
server_key: "a6697252e45d6287254796a0c75062a8"
server_key_iv: "3671d99d163e175695c7b85f095339e1"
server_key_version: "encryptionkey@0"
I20260927 00:55:22.378013 15017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:22.416504 15017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:22.420785 15017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:22.421020 15017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:22.421819 15017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:22.422945 15017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:22.422997 15017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.423055 15017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:22.423079 15017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.434718 15017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36901
I20260927 00:55:22.434743 15146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36901 every 8 connection(s)
I20260927 00:55:22.436156 15017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:55:22.443105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15017
Sep 27 00:55:22 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:22.452476 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.438696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45563 (local address 127.0.98.62:41347)
0927 00:55:22.438890 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.438894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.440048 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 00:55:22.441503 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.441507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.441572 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.441650 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:22.442929 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.443569 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.444980 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.445223 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.445303 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.448311 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:22.448314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:22.448316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:22.448341 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:22.449974 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:22.450541 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:22.450545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:22.450547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:22.450654 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:22.451029 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:22.451033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:22.451035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:22.451306 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:22.451398 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.452116 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.452278 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:55:22.453455 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.444023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.0.98.62:41347)
0927 00:55:22.444306 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.444310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.444468 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:55:22.444630 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.444632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.444683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.444823 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:22.445616 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.446215 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.447319 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.447706 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.447812 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.449544 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:22.449548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:22.449550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:22.449595 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:22.451226 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:22.451742 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:22.451745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:22.451747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:22.451830 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:22.452167 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:22.452170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:22.452171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:22.452373 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:22.452640 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.453180 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.453299 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260927 00:55:22.453464 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.439280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:45563)
0927 00:55:22.439869 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.439924 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:22.441224 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.441848 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.441856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.442254 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:22.442779 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.442793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.443688 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.443691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.444861 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.444873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.445394 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.445399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.445616 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.446195 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:22.446212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:22.448134 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:22.450092 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:22.450098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:22.450115 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:22.450425 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:22.450762 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:22.450764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:22.450766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:22.450921 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:22.451472 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:22.451479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:22.451818 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.452035 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.452367 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":156,"threads_started":1}
I20260927 00:55:22.454802 15147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260927 00:55:22.455067 15147 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:22.455634 15147 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260927 00:55:22.456839 14689 ts_manager.cc:194] Registered new tserver with Master: e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901)
I20260927 00:55:22.457530 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45563
I20260927 00:55:22.468462 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:55:22.473878 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.469268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.0.98.62:41347)
0927 00:55:22.469427 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.469431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.469582 (+ 151us) server_negotiation.cc:407] Connection header received
0927 00:55:22.469772 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.469774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.469917 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.470034 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:55:22.470224 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.470797 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.471815 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.472110 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.472212 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.472719 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:55:22.473345 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:55:22.473455 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.473601 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.473692 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:55:22.474704 14689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:55:22.479012 14689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:55:22.488585 14811 tablet_service.cc:1467] Processing CreateTablet for tablet bf23eac3de764f81aebc6034383dc8b7 (DEFAULT_TABLE table=test-table [id=5e13281775e948748343be770fbd88c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:22.489264 14811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf23eac3de764f81aebc6034383dc8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:22.492211 14769 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap starting.
I20260927 00:55:22.493219 14769 tablet_bootstrap.cc:654] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:22.493572 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.487798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.0.98.3:36901)
0927 00:55:22.488259 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.488273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.488305 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:55:22.488422 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.488428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.488975 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.489170 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:22.489471 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.490532 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.491916 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.492858 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.492992 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.493196 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.493292 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.493407 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":159,"threads_started":1}
I20260927 00:55:22.493557 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.487511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.0.1:50672)
0927 00:55:22.488129 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.488146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:22.488285 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.489157 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.489167 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.489185 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:22.489324 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.489335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.490655 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.490659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.491779 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.491788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.492987 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.492991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.493113 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.493139 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.493244 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.493377 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529}
I20260927 00:55:22.494694 14769 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: No bootstrap required, opened a new log
I20260927 00:55:22.494781 14769 ts_tablet_manager.cc:1397] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:55:22.495241 14769 raft_consensus.cc:348] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.495368 14769 raft_consensus.cc:374] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:22.495416 14769 raft_consensus.cc:729] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3397a2a25fdd4180a0d5d0cb8ee9a354, State: Initialized, Role: FOLLOWER
I20260927 00:55:22.495545 14769 consensus_queue.cc:260] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.495801 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.488338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.0.98.2:45269)
0927 00:55:22.488923 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.488935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.488955 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:55:22.489032 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.489037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.489269 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.489446 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:22.489896 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.491425 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.492962 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.494202 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.494307 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.494474 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.495503 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.495615 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":169,"threads_started":1}
I20260927 00:55:22.495801 14769 ts_tablet_manager.cc:1428] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:55:22.496083 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.487772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.0.1:35764)
0927 00:55:22.488639 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.488652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:22.488752 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.489614 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.489617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.489635 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:22.489735 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.489741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.491584 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.491588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.492820 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.492831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.494894 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.494899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.495023 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.495056 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.495136 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.495269 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":146,"threads_started":1}
I20260927 00:55:22.497812 15081 tablet_service.cc:1467] Processing CreateTablet for tablet bf23eac3de764f81aebc6034383dc8b7 (DEFAULT_TABLE table=test-table [id=5e13281775e948748343be770fbd88c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:22.499308 15081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf23eac3de764f81aebc6034383dc8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:22.499820 14949 tablet_service.cc:1467] Processing CreateTablet for tablet bf23eac3de764f81aebc6034383dc8b7 (DEFAULT_TABLE table=test-table [id=5e13281775e948748343be770fbd88c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:22.500871 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf23eac3de764f81aebc6034383dc8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:22.508183 15160 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Bootstrap starting.
I20260927 00:55:22.508236 15161 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Bootstrap starting.
I20260927 00:55:22.511411 15161 tablet_bootstrap.cc:654] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:22.511408 15160 tablet_bootstrap.cc:654] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:22.512166 15161 log.cc:824] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:22.512370 15160 log.cc:824] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:22.514503 15161 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: No bootstrap required, opened a new log
I20260927 00:55:22.514559 15160 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: No bootstrap required, opened a new log
I20260927 00:55:22.514750 15161 ts_tablet_manager.cc:1397] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:55:22.514763 15160 ts_tablet_manager.cc:1397] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 00:55:22.518157 15160 raft_consensus.cc:348] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.518427 15160 raft_consensus.cc:374] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:22.518481 15160 raft_consensus.cc:729] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d484a69ee7428fa60c59e1f4c04a2d, State: Initialized, Role: FOLLOWER
I20260927 00:55:22.518399 15161 raft_consensus.cc:348] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.518746 15161 raft_consensus.cc:374] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:22.518812 15161 raft_consensus.cc:729] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d074e65621433a8bad3480c13078ab, State: Initialized, Role: FOLLOWER
I20260927 00:55:22.518963 15160 consensus_queue.cc:260] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.519345 15161 consensus_queue.cc:260] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.519868 15147 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260927 00:55:22.520068 15160 ts_tablet_manager.cc:1428] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:55:22.520073 15015 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260927 00:55:22.520504 15161 ts_tablet_manager.cc:1428] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:55:22.636308 14885 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:22.636600 14885 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.637769 14885 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
I20260927 00:55:22.642341 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.638211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.98.1:39959)
0927 00:55:22.638592 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.638679 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:22.638886 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.639260 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.639263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.639285 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:22.639404 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.639410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.640147 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.640151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.641160 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.641170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.641852 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.641855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.641945 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.641970 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.642035 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.642126 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":56,"thread_start_us":137,"threads_started":1}
I20260927 00:55:22.642308 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.638339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39959 (local address 127.0.98.3:36901)
0927 00:55:22.638537 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.638541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.638775 (+ 234us) server_negotiation.cc:407] Connection header received
0927 00:55:22.639075 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.639078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.639134 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.639236 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:22.639507 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.640041 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.641291 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.641736 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.641834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.641909 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.642061 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.642150 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 00:55:22.642318 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.638840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48421 (local address 127.0.98.2:45269)
0927 00:55:22.638980 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.638984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.639137 (+ 153us) server_negotiation.cc:407] Connection header received
0927 00:55:22.639350 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.639354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.639402 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.639469 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:22.639732 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.640225 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.641319 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.641722 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.641814 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.641888 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.642115 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.642186 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:55:22.642359 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.638679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.98.1:48421)
0927 00:55:22.639024 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.639040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:22.639228 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:22.639496 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:22.639499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:22.639516 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:22.639633 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.639639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.640334 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.640338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.641190 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:22.641200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.641888 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.641892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:22.641989 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.642020 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:55:22.642087 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:55:22.642183 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":94,"threads_started":1}
I20260927 00:55:22.643272 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf23eac3de764f81aebc6034383dc8b7" candidate_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d074e65621433a8bad3480c13078ab" is_pre_election: true
I20260927 00:55:22.643392 15101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf23eac3de764f81aebc6034383dc8b7" candidate_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" is_pre_election: true
I20260927 00:55:22.643611 14969 raft_consensus.cc:2454] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3397a2a25fdd4180a0d5d0cb8ee9a354 in term 0.
I20260927 00:55:22.643764 15101 raft_consensus.cc:2454] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3397a2a25fdd4180a0d5d0cb8ee9a354 in term 0.
I20260927 00:55:22.644088 14763 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354, 98d074e65621433a8bad3480c13078ab; no voters:
I20260927 00:55:22.644366 14885 raft_consensus.cc:2781] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:55:22.644459 14885 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:55:22.644511 14885 raft_consensus.cc:3037] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:22.645491 14885 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.645823 14885 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 1 election: Requested vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
I20260927 00:55:22.646237 15101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf23eac3de764f81aebc6034383dc8b7" candidate_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
I20260927 00:55:22.646252 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf23eac3de764f81aebc6034383dc8b7" candidate_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d074e65621433a8bad3480c13078ab"
I20260927 00:55:22.646371 14969 raft_consensus.cc:3037] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:22.646390 15101 raft_consensus.cc:3037] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:22.647468 14969 raft_consensus.cc:2454] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3397a2a25fdd4180a0d5d0cb8ee9a354 in term 1.
I20260927 00:55:22.647774 14763 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354, 98d074e65621433a8bad3480c13078ab; no voters:
I20260927 00:55:22.647979 15101 raft_consensus.cc:2454] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3397a2a25fdd4180a0d5d0cb8ee9a354 in term 1.
I20260927 00:55:22.647991 14885 raft_consensus.cc:2781] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:22.648209 14885 raft_consensus.cc:686] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 LEADER]: Becoming Leader. State: Replica: 3397a2a25fdd4180a0d5d0cb8ee9a354, State: Running, Role: LEADER
I20260927 00:55:22.648533 14885 consensus_queue.cc:237] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:22.651927 14688 catalog_manager.cc:5495] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } health_report { overall_health: HEALTHY } } }
W20260927 00:55:22.687364 15148 tablet.cc:2369] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:55:22.695904 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.691815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.98.1:42213)
0927 00:55:22.692197 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:22.692204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:22.692217 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:55:22.692297 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:22.692299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:22.692355 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:22.692461 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:55:22.692688 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.693602 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.694410 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:22.694636 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:22.694718 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:22.694964 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:55:22.695490 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:55:22.695592 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:22.695665 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:55:22.695724 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260927 00:55:22.701023 14969 raft_consensus.cc:1261] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Refusing update from remote peer 3397a2a25fdd4180a0d5d0cb8ee9a354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:22.701082 15101 raft_consensus.cc:1261] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Refusing update from remote peer 3397a2a25fdd4180a0d5d0cb8ee9a354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:22.701799 14885 consensus_queue.cc:1035] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:22.702049 15168 consensus_queue.cc:1035] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:55:22.704594 15016 tablet.cc:2369] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:55:22.714665 15173 mvcc.cc:205] Tried to move back new op lower bound from 7333767260975853568 to 7333767260771028992. Current Snapshot: MvccSnapshot[applied={T|T < 7333767260975853568 or (T in {7333767260975853568})}]
I20260927 00:55:22.721506 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:22.721752 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:22.724757 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Sending DeleteTablet for 3 replicas of tablet bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:22.725744 14811 tablet_service.cc:1514] Processing DeleteTablet for tablet bf23eac3de764f81aebc6034383dc8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55268
I20260927 00:55:22.725816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14727
I20260927 00:55:22.725997 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet bf23eac3de764f81aebc6034383dc8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35764
I20260927 00:55:22.726619 15182 tablet_replica.cc:330] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: stopping tablet replica
I20260927 00:55:22.726555 15081 tablet_service.cc:1514] Processing DeleteTablet for tablet bf23eac3de764f81aebc6034383dc8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50672
I20260927 00:55:22.727088 15182 raft_consensus.cc:2229] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:22.727543 15183 tablet_replica.cc:330] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: stopping tablet replica
I20260927 00:55:22.727569 15182 raft_consensus.cc:2258] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:22.727911 15183 raft_consensus.cc:2229] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:22.728255 15183 raft_consensus.cc:2258] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:22.729426 15182 ts_tablet_manager.cc:1905] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:22.729986 15183 ts_tablet_manager.cc:1905] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:22.732100 15182 ts_tablet_manager.cc:1918] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:22.732216 15182 log.cc:1196] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:22.732501 15182 ts_tablet_manager.cc:1939] T bf23eac3de764f81aebc6034383dc8b7 P 98d074e65621433a8bad3480c13078ab: Deleting consensus metadata
I20260927 00:55:22.732924 15183 ts_tablet_manager.cc:1918] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:22.733044 15183 log.cc:1196] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:22.733386 15183 ts_tablet_manager.cc:1939] T bf23eac3de764f81aebc6034383dc8b7 P e1d484a69ee7428fa60c59e1f4c04a2d: Deleting consensus metadata
I20260927 00:55:22.734076 14673 catalog_manager.cc:4841] TS 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): tablet bf23eac3de764f81aebc6034383dc8b7 (table test-table [id=5e13281775e948748343be770fbd88c4]) successfully deleted
W20260927 00:55:22.734429 14676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42213 (error 108)
I20260927 00:55:22.734947 14673 catalog_manager.cc:4841] TS e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): tablet bf23eac3de764f81aebc6034383dc8b7 (table test-table [id=5e13281775e948748343be770fbd88c4]) successfully deleted
I20260927 00:55:22.735379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14881
I20260927 00:55:22.735709 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:22.735415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42213 (local address 127.0.0.1:53910)
0927 00:55:22.735562 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:22.735628 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:55:22.736179 14676 catalog_manager.cc:4584] TS 3397a2a25fdd4180a0d5d0cb8ee9a354 (127.0.98.1:42213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf23eac3de764f81aebc6034383dc8b7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42213: connect: Connection refused (error 111)
I20260927 00:55:22.742197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15017
I20260927 00:55:22.748359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 14656
I20260927 00:55:22.755757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--webserver_interface=127.0.98.62
--webserver_port=35825
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:22.867388 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:22.867672 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:22.867724 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:22.871676 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:22.871737 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:22.871762 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:22.871783 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:22.871803 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:22.877086 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:22.878201 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:22.879257 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:22.889079 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:22.889248 15185 server_base.cc:1034] running on GCE node
W20260927 00:55:22.889129 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:22.889079 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:22.890267 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:22.891811 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:22.893006 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790470522892975 us; error 50 us; skew 500 ppm
Sep 27 00:55:22 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:22.896377 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:22.897531 15185 webserver.cc:460] Webserver started at http://127.0.98.62:35825/ using document root <none> and password file <none>
I20260927 00:55:22.898227 15185 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:22.898308 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:22.903352 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:55:22.906289 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:22.908075 15185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:22.908244 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "587fa53c18a54f7a92a21b2ee54c5a7e"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "d97c53c9674bf63588a1914b32ee0bbe"
server_key_iv: "1b7d1d52fbf03aff157010778dd3212a"
server_key_version: "encryptionkey@0"
I20260927 00:55:22.908998 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:22.921836 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:22.926012 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:22.926270 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:22.935612 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41347
I20260927 00:55:22.935926 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41347 every 8 connection(s)
I20260927 00:55:22.937179 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:22.942037 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:55:22.943819 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap starting.
I20260927 00:55:22.943917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15185
I20260927 00:55:22.944410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41347
--webserver_port=39237
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:55:22.966434 15253 log.cc:824] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:22.979933 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:55:22.980330 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e: Bootstrap complete.
I20260927 00:55:22.984337 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:22.984751 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587fa53c18a54f7a92a21b2ee54c5a7e, State: Initialized, Role: FOLLOWER
I20260927 00:55:22.985239 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:22.985375 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:22.985430 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:22.985543 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:55:22.987643 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:22.987967 15253 leader_election.cc:302] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587fa53c18a54f7a92a21b2ee54c5a7e; no voters:
I20260927 00:55:22.988271 15253 leader_election.cc:288] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:55:22.988385 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:55:22.988591 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [term 3 LEADER]: Becoming Leader. State: Replica: 587fa53c18a54f7a92a21b2ee54c5a7e, State: Running, Role: LEADER
I20260927 00:55:22.988901 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260927 00:55:22.989576 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:22.989966 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587fa53c18a54f7a92a21b2ee54c5a7e. Latest consensus state: current_term: 3 leader_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260927 00:55:22.990193 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:22.990264 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587fa53c18a54f7a92a21b2ee54c5a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260927 00:55:22.990404 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587fa53c18a54f7a92a21b2ee54c5a7e [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:22.991031 15262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:22.997641 15262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:22.998332 15262 catalog_manager.cc:1259] Loaded cluster ID: 4f9e0367d07f463c98f1caad8db61906
I20260927 00:55:22.998392 15262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 00:55:22.999333 15262 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbddcb629ad google::LogMessage::Fail() at ??:0
@ 0x7fbddcb6588a google::LogMessage::SendToLog() at ??:0
@ 0x7fbddcb624de google::LogMessage::Flush() at ??:0
@ 0x7fbddcb63a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbde74a1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbde74a2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbde749f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbde74e3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbddedb43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbddedb4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbddedb63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbddeda5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbde03e96db start_thread at ??:0
@ 0x7fbddb66061f clone at ??:0
W20260927 00:55:23.103731 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:23.104072 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:23.104139 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:23.108708 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:23.108803 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:23.108899 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:55:23.115356 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39237
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:23.116858 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:23.118171 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:23.130393 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:23.130379 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:23.130650 15255 server_base.cc:1034] running on GCE node
W20260927 00:55:23.130379 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:23.131438 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:23.132304 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:23.133507 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790470523133481 us; error 52 us; skew 500 ppm
Sep 27 00:55:23 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:23.137046 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:55:23.138386 15255 webserver.cc:460] Webserver started at http://127.0.98.1:39237/ using document root <none> and password file <none>
I20260927 00:55:23.139129 15255 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:23.139256 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:23.145233 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 00:55:23.149147 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.151363 15255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:55:23.151602 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "a22cab90894bf9ed511d8c34b03d2815"
server_key_iv: "1a0d6af1b253589893ad8e675f0160a5"
server_key_version: "encryptionkey@0"
I20260927 00:55:23.152632 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:23.180128 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:23.185163 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:23.185442 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:23.186414 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:23.188220 15311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:55:23.192873 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:55:23.193020 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 00:55:23.193135 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:55:23.196419 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:55:23.196511 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260927 00:55:23.196976 15311 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap starting.
I20260927 00:55:23.210107 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42213
I20260927 00:55:23.210740 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42213 every 8 connection(s)
I20260927 00:55:23.211828 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:55:23.214202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15255
I20260927 00:55:23.214701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41347
--webserver_port=38503
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:55:23.219028 15311 log.cc:824] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:23.228840 15311 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:55:23.229287 15311 tablet_bootstrap.cc:492] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Bootstrap complete.
I20260927 00:55:23.230625 15311 ts_tablet_manager.cc:1397] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.008s
I20260927 00:55:23.234043 15311 raft_consensus.cc:348] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:23.234655 15311 raft_consensus.cc:729] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3397a2a25fdd4180a0d5d0cb8ee9a354, State: Initialized, Role: FOLLOWER
I20260927 00:55:23.235260 15311 consensus_queue.cc:260] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:23.236454 15311 ts_tablet_manager.cc:1428] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 00:55:23.276695 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:55:23.214771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:37203)
0927 00:55:23.215483 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.215548 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:23.216704 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:23.275422 (+ 58718us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":225,"threads_started":1}
W20260927 00:55:23.277487 15419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41347 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41347: Client connection negotiation failed: client connection to 127.0.98.62:41347: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:55:23.278680 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.278242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:60563)
0927 00:55:23.278445 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.278567 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 00:55:23.279729 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.279445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:59931)
0927 00:55:23.279583 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.279648 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 00:55:23.280007 15419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:55:23.355986 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:23.356390 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:23.356443 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:23.361560 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:23.361677 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:23.361768 15422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:55:23.368373 15422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38503
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:23.369915 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:23.371328 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:23.383548 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:23.383567 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:23.383642 15422 server_base.cc:1034] running on GCE node
W20260927 00:55:23.383579 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:23.384371 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:23.385164 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:23.386343 15422 hybrid_clock.cc:648] HybridClock initialized: now 1790470523386328 us; error 34 us; skew 500 ppm
Sep 27 00:55:23 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:23.389596 15422 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:55:23.390671 15422 webserver.cc:460] Webserver started at http://127.0.98.2:38503/ using document root <none> and password file <none>
I20260927 00:55:23.391410 15422 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:23.391487 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:23.396189 15422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:55:23.398850 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.400604 15422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:23.400744 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "98d074e65621433a8bad3480c13078ab"
format_stamp: "Formatted at 2026-09-27 00:55:20 on dist-test-slave-9fh2"
server_key: "ef74ab39132898ba66f810ad6495975a"
server_key_iv: "90562d29e5407ea475be12b001488ffb"
server_key_version: "encryptionkey@0"
I20260927 00:55:23.401379 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:23.423547 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:23.427337 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:23.427536 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:23.428184 15422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:23.429095 15422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:23.429145 15422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.429200 15422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:23.429230 15422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.439234 15422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45269
I20260927 00:55:23.439317 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45269 every 8 connection(s)
I20260927 00:55:23.440384 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:55:23.445469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15422
I20260927 00:55:23.445717 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.443284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:34215)
0927 00:55:23.443991 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.444162 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":143,"threads_started":1}
I20260927 00:55:23.445932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36901
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41347
--webserver_port=40247
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:23.446296 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41347 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41347: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
I20260927 00:55:23.447072 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.446812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:59887)
0927 00:55:23.446944 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.447003 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:55:23.447933 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.447713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:49201)
0927 00:55:23.447828 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.447873 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 00:55:23.448136 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:55:23.576683 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:23.577028 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:23.577077 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:23.581362 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:23.581449 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:23.581518 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:55:23.587800 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40247
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:23.589247 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:23.590546 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:23.602272 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:23.602293 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:23.602448 15556 server_base.cc:1034] running on GCE node
W20260927 00:55:23.602303 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:23.603190 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:23.604053 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:23.605235 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790470523605209 us; error 47 us; skew 500 ppm
Sep 27 00:55:23 dist-test-slave-9fh2 krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:23.608381 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:55:23.609373 15556 webserver.cc:460] Webserver started at http://127.0.98.3:40247/ using document root <none> and password file <none>
I20260927 00:55:23.609956 15556 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:23.610025 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:23.614645 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:55:23.617245 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.618953 15556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:55:23.619096 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "e1d484a69ee7428fa60c59e1f4c04a2d"
format_stamp: "Formatted at 2026-09-27 00:55:21 on dist-test-slave-9fh2"
server_key: "a6697252e45d6287254796a0c75062a8"
server_key_iv: "3671d99d163e175695c7b85f095339e1"
server_key_version: "encryptionkey@0"
I20260927 00:55:23.619757 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:23.642221 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:23.645759 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:23.645959 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:23.646584 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:23.647547 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:23.647598 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.647650 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:23.647679 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:23.657387 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36901
I20260927 00:55:23.657459 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36901 every 8 connection(s)
I20260927 00:55:23.658494 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:55:23.663403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15556
I20260927 00:55:23.663638 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.661250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:45351)
0927 00:55:23.661944 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.662127 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":119,"threads_started":1}
W20260927 00:55:23.664395 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41347 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41347: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
I20260927 00:55:23.665202 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.664883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:49805)
0927 00:55:23.665058 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.665122 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 00:55:23.665985 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:23.665761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:57403)
0927 00:55:23.665882 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:23.665932 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:55:23.666249 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:55:24.281522 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.280897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:54893)
0927 00:55:24.281296 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:24.281388 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
I20260927 00:55:24.449676 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.449026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:42399)
0927 00:55:24.449443 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:24.449535 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":104,"threads_started":1}
I20260927 00:55:24.667699 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.667115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:47931)
0927 00:55:24.667474 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:24.667583 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":102,"threads_started":1}
I20260927 00:55:24.880944 15692 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:24.881152 15692 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:24.882341 15692 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
Sep 27 00:55:24 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:55:24 dist-test-slave-9fh2 krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:55:24.898693 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.883559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36869 (local address 127.0.98.2:45269)
0927 00:55:24.883978 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:24.884003 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:24.884043 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:55:24.884820 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:24.884828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:24.885191 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:24.885531 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:24.886897 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.887972 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.889313 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.889783 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.889938 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:24.893479 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:24.893485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:24.893501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:24.893539 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:24.896018 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:24.896647 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:24.896654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:24.896659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:24.896787 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:24.897064 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:24.897066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:24.897067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:24.897571 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:24.897796 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:24.898189 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 00:55:24.898472 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20260927 00:55:24.898681 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.882753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.98.1:37481)
0927 00:55:24.883398 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:24.883414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:24.883541 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:24.885480 (+ 1939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:24.885487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:24.886146 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:24.886707 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:24.886725 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.888168 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.888171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:24.889178 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:24.889189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.889937 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.889940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:24.890164 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:24.890916 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:24.890943 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:24.893124 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:24.895755 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:24.895763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:24.895779 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:24.896169 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:24.896560 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:24.896563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:24.896564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:24.896748 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:24.897537 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:24.897545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:24.897833 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 00:55:24.898036 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:55:24.898444 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":180,"spinlock_wait_cycles":45696,"thread_start_us":96,"threads_started":1}
I20260927 00:55:24.898690 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.882944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37481 (local address 127.0.98.3:36901)
0927 00:55:24.883298 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:24.883333 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:24.883548 (+ 215us) server_negotiation.cc:407] Connection header received
0927 00:55:24.884550 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:24.884567 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:24.885057 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:24.885370 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:24.886909 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.888015 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.889308 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.889800 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.889979 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:24.893520 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:24.893528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:24.893548 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:24.893593 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:24.895607 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:24.896295 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:24.896301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:24.896308 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:24.896460 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:24.896870 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:24.896873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:24.896874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:24.897410 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:24.897518 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:24.898191 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 00:55:24.898491 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260927 00:55:24.898723 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:24.883196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.98.1:36869)
0927 00:55:24.883527 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:24.883539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:24.883623 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:24.885490 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:24.885495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:24.886137 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:24.886708 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:24.886726 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.888121 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.888125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:24.889176 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:24.889186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:24.889914 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:24.889918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:24.890166 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:24.890914 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:24.890942 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:24.893124 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:24.896175 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:24.896178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:24.896180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:24.896525 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:24.896877 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:24.896880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:24.896882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:24.896973 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:24.897772 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:24.897774 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:24.897890 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 00:55:24.898039 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 00:55:24.898529 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":210,"spinlock_wait_cycles":29312,"thread_start_us":80,"threads_started":1}
W20260927 00:55:24.901882 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
W20260927 00:55:24.902860 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:24.903002 15305 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354; no voters: 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d
I20260927 00:55:24.903478 15692 raft_consensus.cc:2726] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:55:25.283012 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:25.282457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:55581)
0927 00:55:25.282776 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:25.282882 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 00:55:25.451504 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:25.450769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:34415)
0927 00:55:25.451242 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:25.451395 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":176,"threads_started":1}
I20260927 00:55:25.669234 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:25.668662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:36173)
0927 00:55:25.669004 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:25.669094 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
I20260927 00:55:26.284689 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:26.283994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:35061)
0927 00:55:26.284449 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:26.284545 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":89,"threads_started":1}
I20260927 00:55:26.453068 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:26.452455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:47441)
0927 00:55:26.452870 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:26.452946 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":171,"threads_started":1}
I20260927 00:55:26.671033 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:26.670414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:53881)
0927 00:55:26.670784 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:26.670872 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":88,"threads_started":1}
I20260927 00:55:26.744647 15702 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:26.744843 15702 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:26.745332 15702 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
W20260927 00:55:26.746553 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
W20260927 00:55:26.746703 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:26.746764 15305 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354; no voters: 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d
I20260927 00:55:26.746932 15702 raft_consensus.cc:2726] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:55:27.286329 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:27.285693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:52557)
0927 00:55:27.286094 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:27.286192 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
I20260927 00:55:27.454730 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:27.454147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:35975)
0927 00:55:27.454540 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:27.454626 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260927 00:55:27.672483 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:27.671945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:57585)
0927 00:55:27.672288 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:27.672357 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":91,"threads_started":1}
I20260927 00:55:28.288509 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:28.287833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:39719)
0927 00:55:28.288302 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:28.288394 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":110,"threads_started":1}
I20260927 00:55:28.456475 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:28.455834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:52591)
0927 00:55:28.456208 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:28.456362 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260927 00:55:28.674211 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:28.673463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:55655)
0927 00:55:28.673964 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:28.674056 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":91,"threads_started":1}
I20260927 00:55:29.283595 15709 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:29.283802 15709 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:29.284390 15709 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
W20260927 00:55:29.285629 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
W20260927 00:55:29.285823 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:29.285882 15305 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354; no voters: 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d
I20260927 00:55:29.286051 15709 raft_consensus.cc:2726] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:55:29.289928 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:29.289496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:49139)
0927 00:55:29.289766 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:29.289832 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260927 00:55:29.458123 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:29.457500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:35365)
0927 00:55:29.457944 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:29.458014 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":193,"threads_started":1}
I20260927 00:55:29.675819 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:29.675243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:58207)
0927 00:55:29.675619 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:29.675705 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260927 00:55:30.291687 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:30.290978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:39811)
0927 00:55:30.291433 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:30.291575 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":154,"threads_started":1}
I20260927 00:55:30.459704 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:30.459002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:37729)
0927 00:55:30.459439 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:30.459561 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":77,"threads_started":1}
I20260927 00:55:30.677614 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:30.676875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:40601)
0927 00:55:30.677416 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:30.677501 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":106,"threads_started":1}
I20260927 00:55:31.293545 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:31.292887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:52543)
0927 00:55:31.293288 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:31.293375 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260927 00:55:31.461195 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:31.460575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:53883)
0927 00:55:31.460947 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:31.461032 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
I20260927 00:55:31.679404 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:31.678769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:43179)
0927 00:55:31.679137 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:31.679269 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":109,"threads_started":1}
I20260927 00:55:32.183128 15719 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:32.183363 15719 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:32.183949 15719 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
W20260927 00:55:32.184911 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
W20260927 00:55:32.185122 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:32.185175 15305 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354; no voters: 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d
I20260927 00:55:32.185381 15719 raft_consensus.cc:2726] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:55:32.295379 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:32.294684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:58081)
0927 00:55:32.295128 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:32.295252 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":127,"threads_started":1}
I20260927 00:55:32.462776 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:32.462195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:45155)
0927 00:55:32.462545 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:32.462641 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
I20260927 00:55:32.681169 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:32.680606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:48893)
0927 00:55:32.680958 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:32.681021 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260927 00:55:33.297571 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:33.296936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:46695)
0927 00:55:33.297361 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:33.297450 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":106,"threads_started":1}
I20260927 00:55:33.464483 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:33.463891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:45905)
0927 00:55:33.464250 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:33.464348 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260927 00:55:33.683001 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:33.682387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:47327)
0927 00:55:33.682802 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:33.682890 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
I20260927 00:55:34.299510 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:34.298742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:57923)
0927 00:55:34.299123 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:34.299381 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260927 00:55:34.465946 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:34.465400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:40123)
0927 00:55:34.465729 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:34.465816 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":87,"threads_started":1}
I20260927 00:55:34.684787 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:34.684146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:36453)
0927 00:55:34.684554 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:34.684652 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260927 00:55:35.301066 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:35.300491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:34987)
0927 00:55:35.300863 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:35.300945 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":95,"threads_started":1}
I20260927 00:55:35.467512 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:35.466781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:52687)
0927 00:55:35.467259 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:35.467354 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":96,"threads_started":1}
I20260927 00:55:35.686409 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:35.685854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:59115)
0927 00:55:35.686213 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:35.686294 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260927 00:55:36.302863 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:36.302233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:60973)
0927 00:55:36.302622 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:36.302724 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
I20260927 00:55:36.469220 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:36.468572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:39543)
0927 00:55:36.469003 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:36.469090 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":101,"threads_started":1}
I20260927 00:55:36.688076 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:36.687498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:39529)
0927 00:55:36.687861 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:36.687950 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260927 00:55:37.067683 15735 raft_consensus.cc:482] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:37.067992 15735 raft_consensus.cc:504] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1d484a69ee7428fa60c59e1f4c04a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } peers { permanent_uuid: "98d074e65621433a8bad3480c13078ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "3397a2a25fdd4180a0d5d0cb8ee9a354" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42213 } }
I20260927 00:55:37.068603 15735 leader_election.cc:288] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901), 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269)
W20260927 00:55:37.069684 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98d074e65621433a8bad3480c13078ab (127.0.98.2:45269): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
W20260927 00:55:37.069871 15305 leader_election.cc:341] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d484a69ee7428fa60c59e1f4c04a2d (127.0.98.3:36901): Not found: Tablet not found: bf23eac3de764f81aebc6034383dc8b7
I20260927 00:55:37.069936 15305 leader_election.cc:302] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3397a2a25fdd4180a0d5d0cb8ee9a354; no voters: 98d074e65621433a8bad3480c13078ab, e1d484a69ee7428fa60c59e1f4c04a2d
I20260927 00:55:37.070163 15735 raft_consensus.cc:2726] T bf23eac3de764f81aebc6034383dc8b7 P 3397a2a25fdd4180a0d5d0cb8ee9a354 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:55:37.304661 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:37.304017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:37655)
0927 00:55:37.304426 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:37.304522 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260927 00:55:37.471014 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:37.470400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:44349)
0927 00:55:37.470795 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:37.470900 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":193,"threads_started":1}
I20260927 00:55:37.689826 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:37.689224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:59865)
0927 00:55:37.689621 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:37.689705 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260927 00:55:38.306905 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:38.306216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:38699)
0927 00:55:38.306714 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:38.306777 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":139,"threads_started":1}
I20260927 00:55:38.473016 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:38.472447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:52327)
0927 00:55:38.472805 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:38.472891 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260927 00:55:38.669200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15255
I20260927 00:55:38.675457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15422
I20260927 00:55:38.681310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15556
2026-09-27T00:55:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19511 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:55:38 dist-test-slave-9fh2 krb5kdc[15743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:55:38 dist-test-slave-9fh2 krb5kdc[15743](info): set up 2 sockets
Sep 27 00:55:38 dist-test-slave-9fh2 krb5kdc[15743](info): commencing operation
krb5kdc: starting...
W20260927 00:55:39.664291 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:55:39 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:55:39.811628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:39.928819 15756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:39.929100 15756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:39.929140 15756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:39.933247 15756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:55:39.933320 15756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:55:39.933336 15756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:55:39.933348 15756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:55:39.933365 15756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:39.933382 15756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:39.933398 15756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:55:39.933413 15756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:55:39.939200 15756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:39.940409 15756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:39.941468 15756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:39.951980 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:39.951980 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:39.952508 15764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:39.953989 15756 server_base.cc:1034] running on GCE node
I20260927 00:55:39.954841 15756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:39.956467 15756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:39.957813 15756 hybrid_clock.cc:648] HybridClock initialized: now 1790470539957761 us; error 75 us; skew 500 ppm
Sep 27 00:55:39 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:39.961498 15756 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:55:39.962814 15756 webserver.cc:460] Webserver started at http://127.0.98.62:35385/ using document root <none> and password file <none>
I20260927 00:55:39.963773 15756 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:39.963881 15756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:39.964195 15756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:39.966929 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "fb6bb65732d74704a44829f427f1ba7a"
format_stamp: "Formatted at 2026-09-27 00:55:39 on dist-test-slave-9fh2"
server_key: "4c1b3679db604ef7ca410ff7edefef0a"
server_key_iv: "d491d0aee26d17acf7914d5336bbb9ac"
server_key_version: "encryptionkey@0"
I20260927 00:55:39.967620 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb6bb65732d74704a44829f427f1ba7a"
format_stamp: "Formatted at 2026-09-27 00:55:39 on dist-test-slave-9fh2"
server_key: "4c1b3679db604ef7ca410ff7edefef0a"
server_key_iv: "d491d0aee26d17acf7914d5336bbb9ac"
server_key_version: "encryptionkey@0"
I20260927 00:55:39.972196 15756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:55:39.975769 15771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:39.977782 15756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:55:39.977984 15756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "fb6bb65732d74704a44829f427f1ba7a"
format_stamp: "Formatted at 2026-09-27 00:55:39 on dist-test-slave-9fh2"
server_key: "4c1b3679db604ef7ca410ff7edefef0a"
server_key_iv: "d491d0aee26d17acf7914d5336bbb9ac"
server_key_version: "encryptionkey@0"
I20260927 00:55:39.978194 15756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:40.004772 15756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:40.010576 15756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:40.010833 15756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:40.021395 15756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37497
I20260927 00:55:40.021724 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37497 every 8 connection(s)
I20260927 00:55:40.022723 15756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:55:40.026264 15824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:40.030866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15756
I20260927 00:55:40.031033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:55:40.031378 392 external_mini_cluster.cc:1428] Setting key 66311c53f14a64dde06b25ddc7c5c520
I20260927 00:55:40.032624 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap starting.
I20260927 00:55:40.035094 15824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:40.035979 15824 log.cc:824] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:40.038830 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: No bootstrap required, opened a new log
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:40.043262 15824 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:55:40.043612 15824 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:40.043730 15824 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6bb65732d74704a44829f427f1ba7a, State: Initialized, Role: FOLLOWER
I20260927 00:55:40.044196 15824 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:55:40.044332 15824 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:55:40.044411 15824 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:55:40.044507 15824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:40.045732 15824 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:55:40.046111 15824 leader_election.cc:302] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb6bb65732d74704a44829f427f1ba7a; no voters:
I20260927 00:55:40.046511 15824 leader_election.cc:288] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:55:40.046607 15829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:40.046788 15829 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 LEADER]: Becoming Leader. State: Replica: fb6bb65732d74704a44829f427f1ba7a, State: Running, Role: LEADER
I20260927 00:55:40.047242 15829 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:55:40.047923 15824 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:55:40.048475 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb6bb65732d74704a44829f427f1ba7a. Latest consensus state: current_term: 1 leader_uuid: "fb6bb65732d74704a44829f427f1ba7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } } }
I20260927 00:55:40.049335 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:40.049220 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb6bb65732d74704a44829f427f1ba7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } } }
I20260927 00:55:40.049700 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: This master's current role is: LEADER
I20260927 00:55:40.050050 15838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:55:40.050735 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.033284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.98.62:37497)
0927 00:55:40.034024 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.034044 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.034094 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:55:40.034960 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.034983 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.035544 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.035961 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:40.036897 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.038225 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.039567 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.040217 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.040389 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.043881 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:40.043889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:40.043912 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:40.043961 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:40.046271 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.047087 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.047097 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.047107 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.047329 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.047654 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.047657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.047659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.048287 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:40.048533 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.049036 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.049472 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":227,"threads_started":1}
I20260927 00:55:40.053107 15838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:55:40.059499 15838 catalog_manager.cc:1347] Generated new cluster ID: f3fde405d67340bcb982ff1a185551f3
I20260927 00:55:40.059667 15838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:55:40.072193 15838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:55:40.073266 15838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:55:40.081164 15838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Generated new TSK 0
I20260927 00:55:40.082398 15838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:55:40.118923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37497
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:40.232612 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:40.232888 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:40.232932 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:40.237145 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:40.237224 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:40.237287 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:55:40.242877 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:40.244065 15850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:40.245158 15850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:40.256004 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:40.256107 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:40.256060 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:40.256731 15850 server_base.cc:1034] running on GCE node
I20260927 00:55:40.257406 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:40.258335 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:40.261693 15850 hybrid_clock.cc:648] HybridClock initialized: now 1790470540261602 us; error 80 us; skew 500 ppm
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:40.265793 15850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:55:40.267056 15850 webserver.cc:460] Webserver started at http://127.0.98.1:35575/ using document root <none> and password file <none>
I20260927 00:55:40.267835 15850 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:40.267918 15850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:40.268164 15850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:40.270852 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "00081122823f4bee9b08b4a7636221f6"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "7db3da141ce09dab0cb504f9e62c8eaf"
server_key_iv: "195683414cf6c7b00bdd16dbe244357a"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.271667 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00081122823f4bee9b08b4a7636221f6"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "7db3da141ce09dab0cb504f9e62c8eaf"
server_key_iv: "195683414cf6c7b00bdd16dbe244357a"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.276921 15850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:55:40.281015 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.282963 15850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:55:40.283229 15850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "00081122823f4bee9b08b4a7636221f6"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "7db3da141ce09dab0cb504f9e62c8eaf"
server_key_iv: "195683414cf6c7b00bdd16dbe244357a"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.283432 15850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:40.301924 15850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:40.306241 15850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:40.306479 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:40.307279 15850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:40.308347 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:40.308403 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.308468 15850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:40.308493 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.319752 15850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41555
I20260927 00:55:40.319841 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41555 every 8 connection(s)
I20260927 00:55:40.320955 15850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:55:40.326107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15850
I20260927 00:55:40.326258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:55:40.326602 392 external_mini_cluster.cc:1428] Setting key 5799f03e36cab781269f2ed3cc06a485
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:55:40.339581 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.323423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44977 (local address 127.0.98.62:37497)
0927 00:55:40.323602 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.323606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.324843 (+ 1237us) server_negotiation.cc:407] Connection header received
0927 00:55:40.326395 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.326401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.326491 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.326606 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:40.328230 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.328895 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.330487 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.330844 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.330970 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.334783 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:40.334788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:40.334791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:40.334831 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:40.336561 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.337499 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.337504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.337506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.337613 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.338148 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.338151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.338152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.338370 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:40.338459 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.339295 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.339429 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:55:40.340958 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.323910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:44977)
0927 00:55:40.324624 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.324696 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.326082 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.326858 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.326869 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.327363 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:40.328050 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.328070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.329085 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.329089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.330313 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.330332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.331093 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.331100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.331378 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.332267 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:40.332289 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:40.334475 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:40.336782 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:40.336793 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:40.336817 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:40.337290 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:40.337755 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:40.337760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:40.337762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:40.338009 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:40.338492 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:40.338500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:40.338934 (+ 434us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.339233 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.339670 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":162,"threads_started":1}
I20260927 00:55:40.342763 15979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37497
I20260927 00:55:40.343122 15979 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:40.343887 15979 heartbeater.cc:511] Master 127.0.98.62:37497 requested a full tablet report, sending...
I20260927 00:55:40.346191 15786 ts_manager.cc:194] Registered new tserver with Master: 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555)
I20260927 00:55:40.347929 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:55:40.360064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37497
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:40.472517 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:40.472983 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:40.473071 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:40.477473 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:40.477543 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:40.477610 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:55:40.483259 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:40.484338 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:40.485323 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:55:40.495414 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:40.495501 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:40.495637 15984 server_base.cc:1034] running on GCE node
W20260927 00:55:40.495417 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:40.496416 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:40.497262 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:40.498449 15984 hybrid_clock.cc:648] HybridClock initialized: now 1790470540498428 us; error 40 us; skew 500 ppm
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:40.501925 15984 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:55:40.503129 15984 webserver.cc:460] Webserver started at http://127.0.98.2:43473/ using document root <none> and password file <none>
I20260927 00:55:40.503788 15984 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:40.503856 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:40.504046 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:40.506134 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "660ddab072ac486fba80c4d4bc18bad7"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "350b83688314d29f6076284fd874e111"
server_key_iv: "e9f8ef2d42ec767947a3d63e1027dbd0"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.506677 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "660ddab072ac486fba80c4d4bc18bad7"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "350b83688314d29f6076284fd874e111"
server_key_iv: "e9f8ef2d42ec767947a3d63e1027dbd0"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.510957 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:55:40.514055 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.515760 15984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:40.515890 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "660ddab072ac486fba80c4d4bc18bad7"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "350b83688314d29f6076284fd874e111"
server_key_iv: "e9f8ef2d42ec767947a3d63e1027dbd0"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.516007 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:40.544731 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:40.548556 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:40.548753 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:40.549413 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:40.550459 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:40.550524 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.550575 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:40.550608 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.560293 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42611
I20260927 00:55:40.560352 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42611 every 8 connection(s)
I20260927 00:55:40.561244 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:55:40.567770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 15984
I20260927 00:55:40.567891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:55:40.568158 392 external_mini_cluster.cc:1428] Setting key 1f21a942a93ef8b54a5c0265f25ecb3b
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:40.579192 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.563663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58401 (local address 127.0.98.62:37497)
0927 00:55:40.563869 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.563874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.564835 (+ 961us) server_negotiation.cc:407] Connection header received
0927 00:55:40.566344 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.566347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.566414 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.566494 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:40.568060 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.568681 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.570148 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.570468 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.570607 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.574328 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:40.574333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:40.574337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:40.574383 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:40.576301 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.577020 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.577025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.577027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.577142 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.577537 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.577540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.577542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.577871 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:40.577983 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.578773 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.578950 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:55:40.580032 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.564009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:58401)
0927 00:55:40.564562 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.564691 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.566091 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.566719 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.566729 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.567230 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:40.567888 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.567909 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.568809 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.568813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.570017 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.570028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.570681 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.570686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.570902 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.571812 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:40.571949 (+ 137us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:40.574034 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:40.576505 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:40.576512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:40.576527 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:40.576878 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:40.577256 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:40.577258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:40.577260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:40.577434 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:40.578014 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:40.578019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:40.578373 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.578658 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.578998 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260927 00:55:40.581791 16113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37497
I20260927 00:55:40.582096 16113 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:55:40.582845 16113 heartbeater.cc:511] Master 127.0.98.62:37497 requested a full tablet report, sending...
I20260927 00:55:40.584440 15786 ts_manager.cc:194] Registered new tserver with Master: 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611)
I20260927 00:55:40.585048 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:55:40.602861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37497
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:55:40.737845 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:55:40.738224 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:55:40.738276 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:55:40.742771 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:55:40.742874 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:55:40.742945 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:55:40.749315 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:55:40.750808 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:55:40.752170 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:55:40.764182 16118 server_base.cc:1034] running on GCE node
W20260927 00:55:40.764050 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:40.764070 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:55:40.764070 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:55:40.765494 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:55:40.766237 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:55:40.767408 16118 hybrid_clock.cc:648] HybridClock initialized: now 1790470540767389 us; error 52 us; skew 500 ppm
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:55:40.770670 16118 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:55:40.771760 16118 webserver.cc:460] Webserver started at http://127.0.98.3:42521/ using document root <none> and password file <none>
I20260927 00:55:40.772342 16118 fs_manager.cc:359] Metadata directory not provided
I20260927 00:55:40.772410 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:55:40.772616 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:55:40.774771 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "3cbc4dae5160440d82e2dd96f62a095c"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "42db248746cb81798f867634983bb682"
server_key_iv: "763e0eb989afc9aed9ea0cdb4eefdb44"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.775381 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3cbc4dae5160440d82e2dd96f62a095c"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "42db248746cb81798f867634983bb682"
server_key_iv: "763e0eb989afc9aed9ea0cdb4eefdb44"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.779685 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:55:40.782686 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.784286 16118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:55:40.784415 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "3cbc4dae5160440d82e2dd96f62a095c"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "42db248746cb81798f867634983bb682"
server_key_iv: "763e0eb989afc9aed9ea0cdb4eefdb44"
server_key_version: "encryptionkey@0"
I20260927 00:55:40.784524 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:55:40.804430 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:55:40.807922 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:55:40.808156 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:55:40.808759 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:55:40.809723 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:55:40.809772 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.809830 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:55:40.809859 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:55:40.819630 16118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35077
I20260927 00:55:40.819696 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35077 every 8 connection(s)
I20260927 00:55:40.820647 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:55:40.821586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16118
I20260927 00:55:40.821694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:55:40.821915 392 external_mini_cluster.cc:1428] Setting key 68f10ead6ce1ab53a5ac5c1eb2119ca8
Sep 27 00:55:40 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:55:40.836997 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.823114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54667 (local address 127.0.98.62:37497)
0927 00:55:40.823298 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.823302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.824871 (+ 1569us) server_negotiation.cc:407] Connection header received
0927 00:55:40.825840 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.825843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.825900 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.825973 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:40.827224 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.827800 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.828934 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.829145 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.829230 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.832264 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:40.832267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:40.832270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:40.832298 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:40.833901 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.834670 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.834673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.834674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.834770 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.835426 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.835430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.835432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.835707 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:40.835797 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.836683 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.836850 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 00:55:40.838469 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.823739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:54667)
0927 00:55:40.824720 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.824775 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.825632 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.826170 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.826177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.826552 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:55:40.827072 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.827086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.827928 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.827931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.828805 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.828815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.829286 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.829290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.829462 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.830122 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:55:40.830138 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:55:40.832002 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:55:40.834075 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:40.834083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:40.834103 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:40.834508 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:40.834916 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:55:40.834920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:55:40.834921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:55:40.835263 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:55:40.835843 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:55:40.835852 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:55:40.836326 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.836591 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.837198 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":211,"threads_started":1}
I20260927 00:55:40.839936 16247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37497
I20260927 00:55:40.840268 16247 heartbeater.cc:464] Registering TS with master...
I20260927 00:55:40.840955 16247 heartbeater.cc:511] Master 127.0.98.62:37497 requested a full tablet report, sending...
I20260927 00:55:40.842300 15786 ts_manager.cc:194] Registered new tserver with Master: 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
I20260927 00:55:40.842922 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54667
I20260927 00:55:40.849862 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:55:40.860213 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.852141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.0.98.62:37497)
0927 00:55:40.852260 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.852263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.852620 (+ 357us) server_negotiation.cc:407] Connection header received
0927 00:55:40.852758 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.852761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.852806 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.852875 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:55:40.853570 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.854116 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.854946 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.855208 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.855298 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.856875 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:55:40.856878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:55:40.856880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:55:40.856908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:55:40.858442 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.858869 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.858872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.858874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.858953 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:55:40.859225 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:55:40.859228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:55:40.859230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:55:40.859568 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:55:40.859690 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.859928 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.860045 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:55:40.864338 15786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:55:40.868326 15786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:55:40.886287 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.879692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41555 (local address 127.0.0.1:55282)
0927 00:55:40.880296 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.880309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.880380 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.881534 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.881537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.881574 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:40.881723 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.881730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.883351 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.883355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.884601 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.884611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.885553 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.885556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.885702 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.885763 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.885888 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.886064 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":3,"spinlock_wait_cycles":4480,"thread_start_us":133,"threads_started":1}
I20260927 00:55:40.886458 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.879808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.0.98.2:42611)
0927 00:55:40.880841 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.880856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.880886 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:55:40.880986 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.880992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.881287 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.881490 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:40.881830 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.883194 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.884834 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.885784 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.885899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.886072 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.886191 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.886309 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":137,"threads_started":1}
I20260927 00:55:40.886904 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.879477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.0.1:35244)
0927 00:55:40.880105 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.880154 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.880291 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.881491 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.881498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.881527 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:40.881691 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.881698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.883353 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.883357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.884680 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.884691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.885928 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.885932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.886035 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.886067 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.886156 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.886266 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":134,"thread_start_us":88,"threads_started":1}
I20260927 00:55:40.886947 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.880431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35077 (local address 127.0.0.1:42974)
0927 00:55:40.880694 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.880710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.880829 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.881872 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.881875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.881891 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:40.881993 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.881999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.883041 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.883045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.885305 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.885316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.886330 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.886334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.886429 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.886451 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.886496 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.886566 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":855,"thread_start_us":96,"threads_started":1}
I20260927 00:55:40.887382 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.880758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.0.98.1:41555)
0927 00:55:40.881107 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.881113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.881134 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:55:40.881208 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.881212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.881380 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.881872 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:40.882024 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.883184 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.884748 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.885365 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.885523 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.885684 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.886732 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.886877 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":139,"threads_started":1}
I20260927 00:55:40.887463 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.880841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.0.98.3:35077)
0927 00:55:40.881433 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.881445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.881468 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:55:40.881540 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.881545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.881756 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.881904 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:40.882073 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.882926 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.885455 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.886206 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.886856 (+ 650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.887002 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.887058 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.887185 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":157,"threads_started":1}
I20260927 00:55:40.890556 16181 tablet_service.cc:1467] Processing CreateTablet for tablet b5bf81b137724a8f83d1ce44ab19021c (DEFAULT_TABLE table=test-table [id=ad8c3d547121430e867888b923f17dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:40.890556 16047 tablet_service.cc:1467] Processing CreateTablet for tablet b5bf81b137724a8f83d1ce44ab19021c (DEFAULT_TABLE table=test-table [id=ad8c3d547121430e867888b923f17dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:40.890798 15913 tablet_service.cc:1467] Processing CreateTablet for tablet b5bf81b137724a8f83d1ce44ab19021c (DEFAULT_TABLE table=test-table [id=ad8c3d547121430e867888b923f17dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:55:40.891813 16181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5bf81b137724a8f83d1ce44ab19021c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:40.891810 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5bf81b137724a8f83d1ce44ab19021c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:40.892441 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5bf81b137724a8f83d1ce44ab19021c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:55:40.899663 16265 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Bootstrap starting.
I20260927 00:55:40.900251 16264 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Bootstrap starting.
I20260927 00:55:40.900851 16266 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Bootstrap starting.
I20260927 00:55:40.903254 16264 tablet_bootstrap.cc:654] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:40.903252 16266 tablet_bootstrap.cc:654] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:40.904754 16264 log.cc:824] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:40.904827 16266 log.cc:824] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:40.906044 16265 tablet_bootstrap.cc:654] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Neither blocks nor log segments found. Creating new log.
I20260927 00:55:40.906814 16265 log.cc:824] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:55:40.907476 16266 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: No bootstrap required, opened a new log
I20260927 00:55:40.907476 16264 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: No bootstrap required, opened a new log
I20260927 00:55:40.907737 16266 ts_tablet_manager.cc:1397] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260927 00:55:40.907737 16264 ts_tablet_manager.cc:1397] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 00:55:40.908702 16265 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: No bootstrap required, opened a new log
I20260927 00:55:40.908900 16265 ts_tablet_manager.cc:1397] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260927 00:55:40.912225 16265 raft_consensus.cc:348] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.912477 16265 raft_consensus.cc:374] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:40.912536 16265 raft_consensus.cc:729] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660ddab072ac486fba80c4d4bc18bad7, State: Initialized, Role: FOLLOWER
I20260927 00:55:40.912437 16264 raft_consensus.cc:348] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.912626 16266 raft_consensus.cc:348] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.912832 16264 raft_consensus.cc:374] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:40.912843 16266 raft_consensus.cc:374] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:55:40.912895 16266 raft_consensus.cc:729] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00081122823f4bee9b08b4a7636221f6, State: Initialized, Role: FOLLOWER
I20260927 00:55:40.912895 16264 raft_consensus.cc:729] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbc4dae5160440d82e2dd96f62a095c, State: Initialized, Role: FOLLOWER
I20260927 00:55:40.913014 16265 consensus_queue.cc:260] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.913417 16264 consensus_queue.cc:260] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.913417 16266 consensus_queue.cc:260] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.914024 16265 ts_tablet_manager.cc:1428] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 00:55:40.914366 15979 heartbeater.cc:503] Master 127.0.98.62:37497 was elected leader, sending a full tablet report...
I20260927 00:55:40.914368 16113 heartbeater.cc:503] Master 127.0.98.62:37497 was elected leader, sending a full tablet report...
I20260927 00:55:40.914549 16266 ts_tablet_manager.cc:1428] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260927 00:55:40.915298 16247 heartbeater.cc:503] Master 127.0.98.62:37497 was elected leader, sending a full tablet report...
I20260927 00:55:40.915766 16264 ts_tablet_manager.cc:1428] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260927 00:55:40.919770 16272 raft_consensus.cc:482] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:55:40.919967 16272 raft_consensus.cc:504] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.921204 16272 leader_election.cc:288] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
I20260927 00:55:40.925189 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.921578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35081 (local address 127.0.98.3:35077)
0927 00:55:40.921725 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.921731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.922006 (+ 275us) server_negotiation.cc:407] Connection header received
0927 00:55:40.922139 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.922142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.922194 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.922255 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:40.922526 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.923036 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.924227 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.924663 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.924755 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.924821 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.924991 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.925063 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:55:40.925211 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.921504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35077 (local address 127.0.98.1:35081)
0927 00:55:40.921917 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.921930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.922040 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.922274 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.922278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.922296 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:40.922430 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.922436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.923136 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.923140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.924098 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.924108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.924772 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.924776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.924883 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.924904 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.924967 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.925050 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":123,"threads_started":1}
I20260927 00:55:40.925974 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.921748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50635 (local address 127.0.98.2:42611)
0927 00:55:40.921885 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.921890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.922136 (+ 246us) server_negotiation.cc:407] Connection header received
0927 00:55:40.922315 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.922318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.922371 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.922437 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:55:40.922804 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.923451 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.924541 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.925007 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.925106 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.925182 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.925723 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.925823 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:55:40.925989 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.921538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.98.1:50635)
0927 00:55:40.922031 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.922043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:55:40.922170 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:55:40.922456 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:55:40.922459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:55:40.922475 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:55:40.922617 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.922622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.923561 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.923565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.924410 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:55:40.924421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.925129 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.925133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:55:40.925252 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.925279 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:55:40.925703 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 00:55:40.925813 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":177,"threads_started":1}
I20260927 00:55:40.925931 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bf81b137724a8f83d1ce44ab19021c" candidate_uuid: "00081122823f4bee9b08b4a7636221f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbc4dae5160440d82e2dd96f62a095c" is_pre_election: true
I20260927 00:55:40.926244 16201 raft_consensus.cc:2454] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00081122823f4bee9b08b4a7636221f6 in term 0.
I20260927 00:55:40.926832 15869 leader_election.cc:302] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6, 3cbc4dae5160440d82e2dd96f62a095c; no voters:
I20260927 00:55:40.926812 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bf81b137724a8f83d1ce44ab19021c" candidate_uuid: "00081122823f4bee9b08b4a7636221f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660ddab072ac486fba80c4d4bc18bad7" is_pre_election: true
I20260927 00:55:40.927089 16067 raft_consensus.cc:2454] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00081122823f4bee9b08b4a7636221f6 in term 0.
I20260927 00:55:40.927228 16272 raft_consensus.cc:2781] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:55:40.927376 16272 raft_consensus.cc:482] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:55:40.927438 16272 raft_consensus.cc:3037] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:40.928972 16272 raft_consensus.cc:504] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.929508 16272 leader_election.cc:288] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 1 election: Requested vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
I20260927 00:55:40.930042 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bf81b137724a8f83d1ce44ab19021c" candidate_uuid: "00081122823f4bee9b08b4a7636221f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660ddab072ac486fba80c4d4bc18bad7"
I20260927 00:55:40.930178 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bf81b137724a8f83d1ce44ab19021c" candidate_uuid: "00081122823f4bee9b08b4a7636221f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbc4dae5160440d82e2dd96f62a095c"
I20260927 00:55:40.930243 16067 raft_consensus.cc:3037] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:40.930316 16201 raft_consensus.cc:3037] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:55:40.931602 16201 raft_consensus.cc:2454] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00081122823f4bee9b08b4a7636221f6 in term 1.
I20260927 00:55:40.931927 16067 raft_consensus.cc:2454] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00081122823f4bee9b08b4a7636221f6 in term 1.
I20260927 00:55:40.932021 15869 leader_election.cc:302] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6, 3cbc4dae5160440d82e2dd96f62a095c; no voters:
I20260927 00:55:40.932277 16272 raft_consensus.cc:2781] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:55:40.932612 16272 raft_consensus.cc:686] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 LEADER]: Becoming Leader. State: Replica: 00081122823f4bee9b08b4a7636221f6, State: Running, Role: LEADER
I20260927 00:55:40.933058 16272 consensus_queue.cc:237] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:55:40.937376 15788 catalog_manager.cc:5495] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00081122823f4bee9b08b4a7636221f6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "00081122823f4bee9b08b4a7636221f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20260927 00:55:40.953547 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.949860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.0.98.1:41555)
0927 00:55:40.950014 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:55:40.950019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:55:40.950051 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:55:40.950261 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:55:40.950264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:55:40.950325 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:55:40.950423 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:55:40.950675 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.951236 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.952100 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:55:40.952312 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:55:40.952396 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:55:40.952704 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:55:40.953151 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:55:40.953257 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:55:40.953319 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 00:55:40.953385 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:55:40.961247 16201 raft_consensus.cc:1261] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Refusing update from remote peer 00081122823f4bee9b08b4a7636221f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:40.961419 16067 raft_consensus.cc:1261] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Refusing update from remote peer 00081122823f4bee9b08b4a7636221f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:55:40.962224 16272 consensus_queue.cc:1035] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:40.962489 16275 consensus_queue.cc:1035] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:55:40.970662 16278 mvcc.cc:205] Tried to move back new op lower bound from 7333767335767646208 to 7333767335665430528. Current Snapshot: MvccSnapshot[applied={T|T < 7333767335767646208}]
I20260927 00:55:40.971961 16280 mvcc.cc:205] Tried to move back new op lower bound from 7333767335767646208 to 7333767335665430528. Current Snapshot: MvccSnapshot[applied={T|T < 7333767335767646208}]
I20260927 00:55:40.972292 16279 mvcc.cc:205] Tried to move back new op lower bound from 7333767335767646208 to 7333767335665430528. Current Snapshot: MvccSnapshot[applied={T|T < 7333767335767646208}]
I20260927 00:55:40.983069 15785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:40.983392 15785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:55:40.986543 15785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Sending DeleteTablet for 3 replicas of tablet b5bf81b137724a8f83d1ce44ab19021c
I20260927 00:55:40.987771 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bf81b137724a8f83d1ce44ab19021c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260927 00:55:40.987911 16047 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bf81b137724a8f83d1ce44ab19021c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35244
I20260927 00:55:40.988467 16287 tablet_replica.cc:330] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: stopping tablet replica
I20260927 00:55:40.988587 16288 tablet_replica.cc:330] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: stopping tablet replica
I20260927 00:55:40.988898 16181 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bf81b137724a8f83d1ce44ab19021c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42974
I20260927 00:55:40.989048 16288 raft_consensus.cc:2229] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:40.989238 16287 raft_consensus.cc:2229] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:55:40.989485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15850
I20260927 00:55:40.989503 16288 raft_consensus.cc:2258] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:40.991089 16289 tablet_replica.cc:330] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: stopping tablet replica
I20260927 00:55:40.991499 16289 raft_consensus.cc:2229] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:55:40.991853 16289 raft_consensus.cc:2258] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:55:40.992865 16288 ts_tablet_manager.cc:1905] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:40.993634 16289 ts_tablet_manager.cc:1905] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:55:40.996263 16288 ts_tablet_manager.cc:1918] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 00:55:40.996302 15773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41555 (error 108)
I20260927 00:55:40.996394 16288 log.cc:1196] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/b5bf81b137724a8f83d1ce44ab19021c
I20260927 00:55:40.996583 16289 ts_tablet_manager.cc:1918] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:55:40.996615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15984
I20260927 00:55:40.996672 16289 log.cc:1196] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/b5bf81b137724a8f83d1ce44ab19021c
I20260927 00:55:40.996765 16288 ts_tablet_manager.cc:1939] T b5bf81b137724a8f83d1ce44ab19021c P 660ddab072ac486fba80c4d4bc18bad7: Deleting consensus metadata
I20260927 00:55:40.996960 16289 ts_tablet_manager.cc:1939] T b5bf81b137724a8f83d1ce44ab19021c P 3cbc4dae5160440d82e2dd96f62a095c: Deleting consensus metadata
I20260927 00:55:40.998826 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:40.998492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41555 (local address 127.0.0.1:55308)
0927 00:55:40.998645 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:40.998730 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:55:40.998939 15775 catalog_manager.cc:4841] TS 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): tablet b5bf81b137724a8f83d1ce44ab19021c (table test-table [id=ad8c3d547121430e867888b923f17dda]) successfully deleted
W20260927 00:55:40.999025 15773 catalog_manager.cc:4584] TS 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5bf81b137724a8f83d1ce44ab19021c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41555: connect: Connection refused (error 111)
I20260927 00:55:41.002530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 16118
I20260927 00:55:41.004704 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:55:41.004338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.0.1:35256)
0927 00:55:41.004524 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 00:55:41.004604 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 00:55:41.008181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 15756
I20260927 00:56:01.014576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37497
--webserver_interface=127.0.98.62
--webserver_port=35385
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:01.130993 16291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:01.131352 16291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:01.131414 16291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:01.135663 16291 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:56:01.135741 16291 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:56:01.135764 16291 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:56:01.135780 16291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:01.135805 16291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:01.135838 16291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:01.135869 16291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:01.135895 16291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:01.141516 16291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:01.142673 16291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:01.143743 16291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:01.158229 16297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:01.158260 16298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:01.158330 16291 server_base.cc:1034] running on GCE node
W20260927 00:56:01.158260 16300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:01.159337 16291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:01.160595 16291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:01.161806 16291 hybrid_clock.cc:648] HybridClock initialized: now 1790470561161779 us; error 51 us; skew 500 ppm
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:01.168762 16291 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:56:01.169859 16291 webserver.cc:460] Webserver started at http://127.0.98.62:35385/ using document root <none> and password file <none>
I20260927 00:56:01.170459 16291 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:01.170547 16291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:01.175539 16291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:56:01.178330 16307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.180161 16291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:01.180336 16291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "fb6bb65732d74704a44829f427f1ba7a"
format_stamp: "Formatted at 2026-09-27 00:55:39 on dist-test-slave-9fh2"
server_key: "4c1b3679db604ef7ca410ff7edefef0a"
server_key_iv: "d491d0aee26d17acf7914d5336bbb9ac"
server_key_version: "encryptionkey@0"
I20260927 00:56:01.181006 16291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:01.194640 16291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:01.198119 16291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:01.198292 16291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:01.206020 16291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37497
I20260927 00:56:01.206378 16359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37497 every 8 connection(s)
I20260927 00:56:01.207299 16291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:01.213363 16360 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:56:01.213528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16291
I20260927 00:56:01.214092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41555
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37497
--webserver_port=35575
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:56:01.216398 16360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap starting.
I20260927 00:56:01.238216 16360 log.cc:824] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:01.249650 16360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:56:01.250139 16360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap complete.
I20260927 00:56:01.254575 16360 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:01.255038 16360 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6bb65732d74704a44829f427f1ba7a, State: Initialized, Role: FOLLOWER
I20260927 00:56:01.255642 16360 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:01.255780 16360 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:01.255858 16360 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:01.256019 16360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:56:01.258332 16360 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:01.258749 16360 leader_election.cc:302] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb6bb65732d74704a44829f427f1ba7a; no voters:
I20260927 00:56:01.259109 16360 leader_election.cc:288] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:56:01.259279 16364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:56:01.259614 16364 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 LEADER]: Becoming Leader. State: Replica: fb6bb65732d74704a44829f427f1ba7a, State: Running, Role: LEADER
I20260927 00:56:01.260052 16364 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:01.260782 16360 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:01.262173 16366 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb6bb65732d74704a44829f427f1ba7a. Latest consensus state: current_term: 2 leader_uuid: "fb6bb65732d74704a44829f427f1ba7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } } }
I20260927 00:56:01.262353 16366 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:01.262775 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb6bb65732d74704a44829f427f1ba7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } } }
I20260927 00:56:01.263000 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:01.268442 16380 catalog_manager.cc:1259] Loaded cluster ID: f3fde405d67340bcb982ff1a185551f3
I20260927 00:56:01.268543 16380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: loading cluster ID for follower catalog manager: success
I20260927 00:56:01.270958 16380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: acquiring CA information for follower catalog manager: success
I20260927 00:56:01.273034 16380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:56:01.273303 16369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:01.277094 16369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:01.277913 16369 catalog_manager.cc:1259] Loaded cluster ID: f3fde405d67340bcb982ff1a185551f3
I20260927 00:56:01.277999 16369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:01.278975 16369 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:01.279717 16369 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Loaded TSK: 0
I20260927 00:56:01.297515 16369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Generated new TSK 1
I20260927 00:56:01.297988 16369 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:56:01.372520 16362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:01.372931 16362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:01.372987 16362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:01.377774 16362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:01.377887 16362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:01.377960 16362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:01.384691 16362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35575
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:01.386399 16362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:01.387811 16362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:01.399923 16387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:01.399938 16388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:01.400588 16390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:01.400557 16362 server_base.cc:1034] running on GCE node
I20260927 00:56:01.401093 16362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:01.401970 16362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:01.403216 16362 hybrid_clock.cc:648] HybridClock initialized: now 1790470561403190 us; error 44 us; skew 500 ppm
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:01.406699 16362 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:56:01.407929 16362 webserver.cc:460] Webserver started at http://127.0.98.1:35575/ using document root <none> and password file <none>
I20260927 00:56:01.408633 16362 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:01.408735 16362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:01.414373 16362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:01.417821 16397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.419788 16362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:56:01.419968 16362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "00081122823f4bee9b08b4a7636221f6"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "7db3da141ce09dab0cb504f9e62c8eaf"
server_key_iv: "195683414cf6c7b00bdd16dbe244357a"
server_key_version: "encryptionkey@0"
I20260927 00:56:01.420754 16362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:01.441821 16362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:01.445894 16362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:01.446125 16362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:01.446887 16362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:01.448312 16404 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:56:01.453261 16362 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:56:01.453398 16362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 00:56:01.453483 16362 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:56:01.456094 16362 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:56:01.456171 16362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260927 00:56:01.456637 16404 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Bootstrap starting.
I20260927 00:56:01.468760 16362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41555
I20260927 00:56:01.468883 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41555 every 8 connection(s)
I20260927 00:56:01.470039 16362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:01.473711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16362
I20260927 00:56:01.474252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42611
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37497
--webserver_port=43473
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:56:01.475883 16404 log.cc:824] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:01.486382 16404 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:56:01.486886 16404 tablet_bootstrap.cc:492] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Bootstrap complete.
I20260927 00:56:01.488293 16404 ts_tablet_manager.cc:1397] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260927 00:56:01.491504 16404 raft_consensus.cc:348] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:56:01.492100 16404 raft_consensus.cc:729] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00081122823f4bee9b08b4a7636221f6, State: Initialized, Role: FOLLOWER
I20260927 00:56:01.492661 16404 consensus_queue.cc:260] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } }
I20260927 00:56:01.493799 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.472676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:59297)
0927 00:56:01.473585 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:01.473670 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:01.474975 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:01.476731 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:01.476746 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:01.477363 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:01.478047 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.478069 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.479875 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.479880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.481376 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.481396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.483115 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.483120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.483422 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.484315 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:01.484335 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:01.486315 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:01.490122 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.490130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.490149 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.490502 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.490892 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.490895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.490897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.491050 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.491780 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:01.491787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:01.492067 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 00:56:01.492294 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 00:56:01.492686 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":143,"threads_started":1}
I20260927 00:56:01.493831 16404 ts_tablet_manager.cc:1428] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:56:01.495455 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.472723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59297 (local address 127.0.98.62:37497)
0927 00:56:01.473207 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.473241 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.473902 (+ 661us) server_negotiation.cc:407] Connection header received
0927 00:56:01.475656 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.475671 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.476113 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.476546 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:01.478325 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.479584 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.481596 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.482469 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.482675 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.487706 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:01.487714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:01.487736 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:01.487782 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:01.489962 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.490636 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.490642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.490646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.490795 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.491178 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.491182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.491184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.491652 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:01.491788 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.494332 (+ 2544us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.494651 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":146,"threads_started":1}
I20260927 00:56:01.498039 16512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37497
I20260927 00:56:01.498347 16512 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:01.498971 16512 heartbeater.cc:511] Master 127.0.98.62:37497 requested a full tablet report, sending...
I20260927 00:56:01.502429 16324 ts_manager.cc:194] Registered new tserver with Master: 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555)
I20260927 00:56:01.506901 16324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59297
I20260927 00:56:01.510423 16512 heartbeater.cc:503] Master 127.0.98.62:37497 was elected leader, sending a full tablet report...
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:56:01.519942 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.506409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.0.98.1:41555)
0927 00:56:01.507613 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.507638 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.507676 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:56:01.507786 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.507793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.508045 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.508261 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:01.509146 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.510223 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.511643 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.512225 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.512368 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.515511 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:01.515524 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:01.515539 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:01.515571 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:01.517555 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.518193 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.518198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.518200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.518287 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.518580 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.518583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.518584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.518921 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:01.519061 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.519528 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.519748 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":139,"threads_started":1}
I20260927 00:56:01.519942 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.506292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41555 (local address 127.0.0.1:59290)
0927 00:56:01.506929 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:01.507000 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:01.507287 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:01.508277 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:01.508283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:01.508777 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:01.508960 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.508968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.510389 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.510393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.511461 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.511475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.512371 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.512402 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.512544 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.513287 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:01.513304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:01.515309 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:01.517732 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.517746 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.517752 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.518052 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.518380 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.518383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.518385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.518481 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.519051 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:01.519058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:01.519263 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 00:56:01.519403 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:56:01.519708 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":214,"threads_started":1}
I20260927 00:56:01.521360 16446 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bf81b137724a8f83d1ce44ab19021c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:55:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59290
I20260927 00:56:01.522415 16521 tablet_replica.cc:330] stopping tablet replica
I20260927 00:56:01.522799 16521 raft_consensus.cc:2229] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:01.523110 16521 raft_consensus.cc:2258] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:01.524358 16521 ts_tablet_manager.cc:1905] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:01.529418 16521 ts_tablet_manager.cc:1918] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:01.529570 16521 log.cc:1196] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal/wals/b5bf81b137724a8f83d1ce44ab19021c
I20260927 00:56:01.529814 16521 ts_tablet_manager.cc:1939] T b5bf81b137724a8f83d1ce44ab19021c P 00081122823f4bee9b08b4a7636221f6: Deleting consensus metadata
I20260927 00:56:01.532379 16309 catalog_manager.cc:4841] TS 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555): tablet b5bf81b137724a8f83d1ce44ab19021c (table test-table [id=ad8c3d547121430e867888b923f17dda]) successfully deleted
W20260927 00:56:01.610003 16516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:01.610304 16516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:01.610348 16516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:01.614799 16516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:01.614902 16516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:01.614984 16516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:01.620651 16516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43473
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:01.621840 16516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:01.623311 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:01.634213 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:01.634230 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:01.634303 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:01.634617 16516 server_base.cc:1034] running on GCE node
I20260927 00:56:01.635241 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:01.636127 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:01.637356 16516 hybrid_clock.cc:648] HybridClock initialized: now 1790470561637318 us; error 62 us; skew 500 ppm
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:01.640898 16516 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:56:01.642047 16516 webserver.cc:460] Webserver started at http://127.0.98.2:43473/ using document root <none> and password file <none>
I20260927 00:56:01.642756 16516 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:01.642839 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:01.647965 16516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:56:01.651084 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.653206 16516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:56:01.653385 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "660ddab072ac486fba80c4d4bc18bad7"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "350b83688314d29f6076284fd874e111"
server_key_iv: "e9f8ef2d42ec767947a3d63e1027dbd0"
server_key_version: "encryptionkey@0"
I20260927 00:56:01.654145 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:01.682511 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:01.687005 16516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:01.687320 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:01.688241 16516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:01.689354 16516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:01.689407 16516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.689468 16516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:01.689495 16516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.700901 16516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42611
I20260927 00:56:01.700955 16649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42611 every 8 connection(s)
I20260927 00:56:01.702280 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:01.703667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16516
I20260927 00:56:01.704258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35077
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37497
--webserver_port=42521
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:01.719411 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.704825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42761 (local address 127.0.98.62:37497)
0927 00:56:01.704963 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.704968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.706895 (+ 1927us) server_negotiation.cc:407] Connection header received
0927 00:56:01.708061 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.708065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.708145 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.708228 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:01.709724 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.710321 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.711601 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.711860 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.711956 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.714998 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:01.715002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:01.715004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:01.715038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:01.716673 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.717400 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.717404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.717405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.717492 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.717855 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.717859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.717861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.718091 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:01.718195 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.718975 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.719206 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:56:01.720422 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.705427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:42761)
0927 00:56:01.706673 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:01.706736 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:01.707800 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:01.708455 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:01.708465 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:01.708944 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:01.709548 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.709564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.710450 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.710453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.711478 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.711490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.712022 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.712026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.712211 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.712990 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:01.713009 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:01.714783 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:01.716901 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.716909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.716934 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.717279 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.717585 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.717588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.717590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.717760 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.718207 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:01.718216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:01.718576 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 00:56:01.718792 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 00:56:01.719298 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":766,"threads_started":1}
I20260927 00:56:01.721976 16650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37497
I20260927 00:56:01.722270 16650 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:01.722913 16650 heartbeater.cc:511] Master 127.0.98.62:37497 requested a full tablet report, sending...
I20260927 00:56:01.724318 16324 ts_manager.cc:194] Registered new tserver with Master: 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611)
I20260927 00:56:01.725116 16324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42761
W20260927 00:56:01.817273 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:01.817579 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:01.817623 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:01.821808 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:01.821895 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:01.821956 16652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:01.827510 16652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42521
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:01.828569 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:01.829545 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:01.840291 16658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:01.840271 16659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:01.840623 16652 server_base.cc:1034] running on GCE node
W20260927 00:56:01.840271 16661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:01.841297 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:01.841997 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:01.843225 16652 hybrid_clock.cc:648] HybridClock initialized: now 1790470561843190 us; error 61 us; skew 500 ppm
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:01.846339 16652 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:56:01.847426 16652 webserver.cc:460] Webserver started at http://127.0.98.3:42521/ using document root <none> and password file <none>
I20260927 00:56:01.848052 16652 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:01.848137 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:01.852662 16652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:01.855235 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.856729 16652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:56:01.856858 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "3cbc4dae5160440d82e2dd96f62a095c"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "42db248746cb81798f867634983bb682"
server_key_iv: "763e0eb989afc9aed9ea0cdb4eefdb44"
server_key_version: "encryptionkey@0"
I20260927 00:56:01.857467 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:01.889313 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:01.893015 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:01.893227 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:01.893832 16652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:01.894698 16652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:01.894745 16652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.894797 16652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:01.894825 16652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:01.904479 16652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35077
I20260927 00:56:01.904552 16781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35077 every 8 connection(s)
I20260927 00:56:01.905586 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:01.913128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16652
Sep 27 00:56:01 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:01.922801 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.914051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.98.62:37497)
0927 00:56:01.914354 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.914358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.914372 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:56:01.914500 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.914503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.914546 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.914617 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:01.915409 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.915964 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.916889 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.917177 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.917266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.918757 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:01.918760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:01.918762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:01.918801 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:01.920564 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.921188 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.921192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.921194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.921288 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.921621 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.921623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.921625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.921845 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:01.922105 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.922294 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.922436 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260927 00:56:01.922854 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.907995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39135 (local address 127.0.98.62:37497)
0927 00:56:01.908207 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.908211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.909190 (+ 979us) server_negotiation.cc:407] Connection header received
0927 00:56:01.910813 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.910816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.910879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.910989 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:01.912697 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.913250 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.914786 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.915031 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.915129 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.918096 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:01.918100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:01.918102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:01.918131 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:01.919676 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.920639 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.920644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.920646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.920876 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.921340 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.921344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.921346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.921795 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:01.921884 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.922579 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.922710 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:56:01.924057 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.908366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:39135)
0927 00:56:01.908962 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:01.909026 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:01.910519 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:01.911324 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:01.911334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:01.911879 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:01.912510 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.912527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.913403 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.913407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.914632 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.914644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.915233 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.915237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.915438 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.916106 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:01.916125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:01.917871 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:01.919871 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.919881 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.919905 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.920370 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.921006 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:01.921008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:01.921009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:01.921197 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:01.921908 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:01.921916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:01.922275 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 00:56:01.922507 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 00:56:01.922909 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":125,"threads_started":1}
I20260927 00:56:01.925339 16782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37497
I20260927 00:56:01.925693 16782 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:01.926378 16782 heartbeater.cc:511] Master 127.0.98.62:37497 requested a full tablet report, sending...
I20260927 00:56:01.927830 16324 ts_manager.cc:194] Registered new tserver with Master: 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
I20260927 00:56:01.928550 16324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39135
I20260927 00:56:01.938235 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:01.942879 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.939199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:37497)
0927 00:56:01.939380 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.939385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.939669 (+ 284us) server_negotiation.cc:407] Connection header received
0927 00:56:01.939866 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.939868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.939982 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.940106 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:01.940355 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.940882 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.941742 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.941950 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.942041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.942333 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:01.942481 (+ 148us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 00:56:01.942738 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55614: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:56:01.942958 16514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55614: token expired
W20260927 00:56:01.943544 16253 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37497)
I20260927 00:56:01.951989 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.944051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.0.98.62:37497)
0927 00:56:01.944245 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.944249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.944266 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:56:01.944315 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.944317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.944370 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.944446 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:01.945115 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.945695 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.946600 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.946836 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.946928 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.948414 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:01.948417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:01.948420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:01.948445 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:01.949988 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.950512 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.950515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.950516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.950598 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:01.950881 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:01.950885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:01.950886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:01.951112 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:01.951360 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.951654 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.951795 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:56:01.955276 16324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:01.959451 16324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:01.969527 16446 tablet_service.cc:1467] Processing CreateTablet for tablet 4bede60cedcb47a89519b105c7b41d1a (DEFAULT_TABLE table=test-table [id=1c8ed9329faf4b099ba31f04de8e6abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:01.970404 16446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bede60cedcb47a89519b105c7b41d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:01.973783 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.968263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.0.1:44246)
0927 00:56:01.968417 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:01.968437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:01.968540 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:01.969625 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:01.969629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:01.969650 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:01.969791 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.969798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.971232 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.971235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.972129 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.972139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.973331 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.973335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.973436 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.973499 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 00:56:01.973548 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:56:01.973636 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 00:56:01.973811 16404 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Bootstrap starting.
I20260927 00:56:01.974628 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.968488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.0.98.2:42611)
0927 00:56:01.969032 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.969049 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.969078 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:56:01.969171 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.969177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.969456 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.969595 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:01.969918 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.970993 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.972257 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.973016 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.974162 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.974323 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.974376 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.974478 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":171,"threads_started":1}
I20260927 00:56:01.974764 16404 tablet_bootstrap.cc:654] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:01.975807 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.968752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35077 (local address 127.0.0.1:49472)
0927 00:56:01.969040 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:01.969055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:01.969155 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:01.971000 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:01.971003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:01.971023 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:01.971139 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.971167 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.972531 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.972535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.973755 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:01.973763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.975289 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.975293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:01.975424 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.975458 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:56:01.975543 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:56:01.975648 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
I20260927 00:56:01.976404 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:01.969883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.98.3:35077)
0927 00:56:01.970314 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:01.970326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:01.970354 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:56:01.970443 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:01.970449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:01.970686 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:01.970857 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:01.971261 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.972404 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.974133 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:01.975116 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:01.975275 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:01.975415 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:01.976086 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 00:56:01.976244 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":171,"threads_started":1}
I20260927 00:56:01.976616 16404 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: No bootstrap required, opened a new log
I20260927 00:56:01.976720 16404 ts_tablet_manager.cc:1397] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:56:01.977263 16404 raft_consensus.cc:348] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:01.977416 16404 raft_consensus.cc:374] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:01.977459 16404 raft_consensus.cc:729] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00081122823f4bee9b08b4a7636221f6, State: Initialized, Role: FOLLOWER
I20260927 00:56:01.977599 16404 consensus_queue.cc:260] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:01.977905 16404 ts_tablet_manager.cc:1428] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:56:01.977873 16584 tablet_service.cc:1467] Processing CreateTablet for tablet 4bede60cedcb47a89519b105c7b41d1a (DEFAULT_TABLE table=test-table [id=1c8ed9329faf4b099ba31f04de8e6abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:01.978971 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bede60cedcb47a89519b105c7b41d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:01.980340 16716 tablet_service.cc:1467] Processing CreateTablet for tablet 4bede60cedcb47a89519b105c7b41d1a (DEFAULT_TABLE table=test-table [id=1c8ed9329faf4b099ba31f04de8e6abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:01.981572 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bede60cedcb47a89519b105c7b41d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:01.989923 16795 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Bootstrap starting.
I20260927 00:56:01.991947 16796 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Bootstrap starting.
I20260927 00:56:01.992939 16795 tablet_bootstrap.cc:654] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:01.993804 16795 log.cc:824] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:01.994710 16796 tablet_bootstrap.cc:654] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:01.995836 16795 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: No bootstrap required, opened a new log
I20260927 00:56:01.996042 16795 ts_tablet_manager.cc:1397] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:01.996061 16796 log.cc:824] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:01.998662 16796 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: No bootstrap required, opened a new log
I20260927 00:56:01.998917 16796 ts_tablet_manager.cc:1397] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 00:56:01.999434 16795 raft_consensus.cc:348] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:01.999764 16795 raft_consensus.cc:374] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:01.999820 16795 raft_consensus.cc:729] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660ddab072ac486fba80c4d4bc18bad7, State: Initialized, Role: FOLLOWER
I20260927 00:56:02.000298 16795 consensus_queue.cc:260] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:02.001021 16650 heartbeater.cc:503] Master 127.0.98.62:37497 was elected leader, sending a full tablet report...
I20260927 00:56:02.002385 16795 ts_tablet_manager.cc:1428] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:56:02.003049 16796 raft_consensus.cc:348] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:02.003489 16796 raft_consensus.cc:374] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:02.003561 16796 raft_consensus.cc:729] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbc4dae5160440d82e2dd96f62a095c, State: Initialized, Role: FOLLOWER
I20260927 00:56:02.004154 16796 consensus_queue.cc:260] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:02.005041 16782 heartbeater.cc:503] Master 127.0.98.62:37497 was elected leader, sending a full tablet report...
I20260927 00:56:02.005568 16796 ts_tablet_manager.cc:1428] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260927 00:56:02.070752 16800 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:02.071004 16800 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:02.072315 16800 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555)
I20260927 00:56:02.076835 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.072810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53371 (local address 127.0.98.2:42611)
0927 00:56:02.072946 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:02.072951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:02.072966 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:56:02.073024 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:02.073026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:02.073075 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:02.073162 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:02.073608 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.074239 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.075620 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.076160 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.076246 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:02.076316 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:02.076646 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:56:02.076723 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:56:02.076890 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.072543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.98.3:53371)
0927 00:56:02.072686 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:02.072703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:02.072807 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:02.073213 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:02.073218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:02.073247 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:02.073409 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:02.073417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.074377 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.074380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:02.075471 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:02.075481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.076321 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.076325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:02.076455 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:02.076488 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:56:02.076590 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:56:02.076689 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":24}
I20260927 00:56:02.077636 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.072705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41555 (local address 127.0.98.3:38059)
0927 00:56:02.073105 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:02.073119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:02.073210 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:02.073613 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:02.073616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:02.073647 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:02.073791 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:02.073799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.074811 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.074814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:02.075890 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:02.075902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.077155 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.077159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:02.077308 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:02.077328 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:56:02.077415 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:56:02.077492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":24,"thread_start_us":154,"threads_started":1}
I20260927 00:56:02.077821 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.072833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38059 (local address 127.0.98.1:41555)
0927 00:56:02.073134 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:02.073139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:02.073183 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:56:02.073324 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:02.073327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:02.073405 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:02.073539 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:02.073978 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.074672 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.076074 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.076897 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.077014 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:02.077229 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:02.077495 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 00:56:02.077662 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260927 00:56:02.077895 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bede60cedcb47a89519b105c7b41d1a" candidate_uuid: "3cbc4dae5160440d82e2dd96f62a095c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660ddab072ac486fba80c4d4bc18bad7" is_pre_election: true
I20260927 00:56:02.078334 16604 raft_consensus.cc:2454] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbc4dae5160440d82e2dd96f62a095c in term 0.
I20260927 00:56:02.078449 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bede60cedcb47a89519b105c7b41d1a" candidate_uuid: "3cbc4dae5160440d82e2dd96f62a095c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00081122823f4bee9b08b4a7636221f6" is_pre_election: true
I20260927 00:56:02.078835 16466 raft_consensus.cc:2454] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbc4dae5160440d82e2dd96f62a095c in term 0.
I20260927 00:56:02.078956 16671 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbc4dae5160440d82e2dd96f62a095c, 660ddab072ac486fba80c4d4bc18bad7; no voters:
I20260927 00:56:02.079365 16800 raft_consensus.cc:2781] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:02.079461 16800 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:02.079509 16800 raft_consensus.cc:3037] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:02.080644 16800 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:02.081027 16800 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [CANDIDATE]: Term 1 election: Requested vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555)
I20260927 00:56:02.081514 16466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bede60cedcb47a89519b105c7b41d1a" candidate_uuid: "3cbc4dae5160440d82e2dd96f62a095c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00081122823f4bee9b08b4a7636221f6"
I20260927 00:56:02.081697 16466 raft_consensus.cc:3037] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:02.082733 16466 raft_consensus.cc:2454] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbc4dae5160440d82e2dd96f62a095c in term 1.
I20260927 00:56:02.083031 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bede60cedcb47a89519b105c7b41d1a" candidate_uuid: "3cbc4dae5160440d82e2dd96f62a095c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660ddab072ac486fba80c4d4bc18bad7"
I20260927 00:56:02.083122 16670 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6, 3cbc4dae5160440d82e2dd96f62a095c; no voters:
I20260927 00:56:02.083237 16604 raft_consensus.cc:3037] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:02.083374 16800 raft_consensus.cc:2781] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:02.083779 16800 raft_consensus.cc:686] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 1 LEADER]: Becoming Leader. State: Replica: 3cbc4dae5160440d82e2dd96f62a095c, State: Running, Role: LEADER
I20260927 00:56:02.084247 16800 consensus_queue.cc:237] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:02.084425 16604 raft_consensus.cc:2454] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbc4dae5160440d82e2dd96f62a095c in term 1.
I20260927 00:56:02.091372 16323 catalog_manager.cc:5495] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3cbc4dae5160440d82e2dd96f62a095c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } health_report { overall_health: UNKNOWN } } }
I20260927 00:56:02.104569 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.100660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.98.3:35077)
0927 00:56:02.100834 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:02.100839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:02.100860 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:56:02.100970 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:02.100973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:02.101042 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:02.101150 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:02.101409 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.102033 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.102891 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:02.103110 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:02.103234 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:02.103557 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:02.104133 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:02.104251 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:02.104318 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 00:56:02.104393 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:56:02.112370 16466 raft_consensus.cc:1261] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Refusing update from remote peer 3cbc4dae5160440d82e2dd96f62a095c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:02.112485 16604 raft_consensus.cc:1261] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Refusing update from remote peer 3cbc4dae5160440d82e2dd96f62a095c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:02.113107 16800 consensus_queue.cc:1035] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [LEADER]: Connected to new peer: Peer: permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:02.113354 16803 consensus_queue.cc:1035] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [LEADER]: Connected to new peer: Peer: permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:02.120913 16806 mvcc.cc:205] Tried to move back new op lower bound from 7333767422402351104 to 7333767422300884992. Current Snapshot: MvccSnapshot[applied={T|T < 7333767422402351104}]
I20260927 00:56:02.121768 16808 mvcc.cc:205] Tried to move back new op lower bound from 7333767422402351104 to 7333767422300884992. Current Snapshot: MvccSnapshot[applied={T|T < 7333767422402351104}]
I20260927 00:56:02.131836 16324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:02.132081 16324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:02.135280 16324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Sending DeleteTablet for 3 replicas of tablet 4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:02.136430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 16362
I20260927 00:56:02.136394 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bede60cedcb47a89519b105c7b41d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44246
I20260927 00:56:02.136516 16446 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bede60cedcb47a89519b105c7b41d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59290
I20260927 00:56:02.137727 16816 tablet_replica.cc:330] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: stopping tablet replica
I20260927 00:56:02.138111 16816 raft_consensus.cc:2229] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:02.138489 16816 raft_consensus.cc:2258] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:02.137163 16716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bede60cedcb47a89519b105c7b41d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49472
I20260927 00:56:02.139505 16817 tablet_replica.cc:330] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: stopping tablet replica
I20260927 00:56:02.139794 16817 raft_consensus.cc:2229] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:56:02.140209 16817 raft_consensus.cc:2258] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:02.141666 16816 ts_tablet_manager.cc:1905] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:02.141773 16817 ts_tablet_manager.cc:1905] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:02.144757 16817 ts_tablet_manager.cc:1918] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 00:56:02.145066 16309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41555 (error 108)
I20260927 00:56:02.145712 16817 log.cc:1196] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:02.145816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 16516
I20260927 00:56:02.147593 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.147241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41555 (local address 127.0.0.1:59304)
0927 00:56:02.147433 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:02.147505 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 00:56:02.148062 16817 ts_tablet_manager.cc:1939] T 4bede60cedcb47a89519b105c7b41d1a P 3cbc4dae5160440d82e2dd96f62a095c: Deleting consensus metadata
W20260927 00:56:02.147774 16309 catalog_manager.cc:4584] TS 00081122823f4bee9b08b4a7636221f6 (127.0.98.1:41555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bede60cedcb47a89519b105c7b41d1a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41555: connect: Connection refused (error 111)
I20260927 00:56:02.149315 16311 catalog_manager.cc:4841] TS 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): tablet 4bede60cedcb47a89519b105c7b41d1a (table test-table [id=1c8ed9329faf4b099ba31f04de8e6abd]) successfully deleted
I20260927 00:56:02.153861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 16652
I20260927 00:56:02.154161 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:02.153913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.0.1:44250)
0927 00:56:02.154024 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:02.154082 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:56:02.160516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 16291
I20260927 00:56:22.168964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37497
--webserver_interface=127.0.98.62
--webserver_port=35385
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:22.300800 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:22.301156 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:22.301230 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:22.305653 16819 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:56:22.305747 16819 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:56:22.305770 16819 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:56:22.305787 16819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:22.305814 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:22.305838 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:22.305869 16819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:22.305899 16819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:22.311873 16819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:22.313284 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:22.314555 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:22.325076 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:22.325201 16819 server_base.cc:1034] running on GCE node
W20260927 00:56:22.325076 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:22.325176 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:22.326200 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:22.327785 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:22.328984 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790470582328964 us; error 38 us; skew 500 ppm
Sep 27 00:56:22 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:22.337802 16819 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:56:22.339309 16819 webserver.cc:460] Webserver started at http://127.0.98.62:35385/ using document root <none> and password file <none>
I20260927 00:56:22.340080 16819 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:22.340169 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:22.345700 16819 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 00:56:22.349084 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:22.351066 16819 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 00:56:22.351341 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "fb6bb65732d74704a44829f427f1ba7a"
format_stamp: "Formatted at 2026-09-27 00:55:39 on dist-test-slave-9fh2"
server_key: "4c1b3679db604ef7ca410ff7edefef0a"
server_key_iv: "d491d0aee26d17acf7914d5336bbb9ac"
server_key_version: "encryptionkey@0"
I20260927 00:56:22.352156 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:22.363410 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:22.367756 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:22.367985 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:22.375530 16819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37497
I20260927 00:56:22.375991 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37497 every 8 connection(s)
I20260927 00:56:22.376751 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:22.377293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16819
I20260927 00:56:22.377775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41555
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37497
--webserver_port=35575
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:56:22.383293 16887 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:56:22.385532 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap starting.
I20260927 00:56:22.407852 16887 log.cc:824] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:22.420529 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:56:22.420993 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Bootstrap complete.
I20260927 00:56:22.425808 16887 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:22.426345 16887 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6bb65732d74704a44829f427f1ba7a, State: Initialized, Role: FOLLOWER
I20260927 00:56:22.427017 16887 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:22.427285 16887 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:22.427410 16887 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:22.427533 16887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:56:22.430358 16887 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:22.430835 16887 leader_election.cc:302] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb6bb65732d74704a44829f427f1ba7a; no voters:
I20260927 00:56:22.431298 16887 leader_election.cc:288] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:56:22.431427 16891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:56:22.431650 16891 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [term 3 LEADER]: Becoming Leader. State: Replica: fb6bb65732d74704a44829f427f1ba7a, State: Running, Role: LEADER
I20260927 00:56:22.432047 16891 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } }
I20260927 00:56:22.432982 16887 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:22.433845 16892 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb6bb65732d74704a44829f427f1ba7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } } }
I20260927 00:56:22.434088 16892 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:22.434571 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb6bb65732d74704a44829f427f1ba7a. Latest consensus state: current_term: 3 leader_uuid: "fb6bb65732d74704a44829f427f1ba7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb6bb65732d74704a44829f427f1ba7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37497 } } }
I20260927 00:56:22.434738 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:22.435374 16900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:22.441102 16900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:22.441725 16900 catalog_manager.cc:1259] Loaded cluster ID: f3fde405d67340bcb982ff1a185551f3
I20260927 00:56:22.441780 16900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:22.443958 16900 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:22.444620 16900 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Loaded TSK: 0
I20260927 00:56:22.444691 16900 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb6bb65732d74704a44829f427f1ba7a: Loaded TSK: 1
F20260927 00:56:22.445420 16900 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1455f339ad google::LogMessage::Fail() at ??:0
@ 0x7f1455f3688a google::LogMessage::SendToLog() at ??:0
@ 0x7f1455f334de google::LogMessage::Flush() at ??:0
@ 0x7f1455f34a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1458bf718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1458beff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f14608987bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f146087299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1460873178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f14608b4df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f14608d382b std::function<>::operator()() at ??:0
@ 0x7f1460872b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1460873f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1460870579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f14608b4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f14581853d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1458185c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f14581873f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1458176512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f14597ba6db start_thread at ??:0
@ 0x7f1454a3161f clone at ??:0
W20260927 00:56:22.528679 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:22.529030 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:22.529101 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:22.535060 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:22.535132 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:22.535241 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:22.543180 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35575
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:22.544665 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:22.546088 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:22.557516 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:22.557622 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:22.557516 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:22.558293 16889 server_base.cc:1034] running on GCE node
I20260927 00:56:22.558940 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:22.560036 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:22.561268 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790470582561223 us; error 58 us; skew 500 ppm
Sep 27 00:56:22 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:22.565541 16889 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:56:22.567073 16889 webserver.cc:460] Webserver started at http://127.0.98.1:35575/ using document root <none> and password file <none>
I20260927 00:56:22.568032 16889 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:22.568135 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:22.574769 16889 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:56:22.578593 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:22.580777 16889 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:56:22.580992 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "00081122823f4bee9b08b4a7636221f6"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "7db3da141ce09dab0cb504f9e62c8eaf"
server_key_iv: "195683414cf6c7b00bdd16dbe244357a"
server_key_version: "encryptionkey@0"
I20260927 00:56:22.581991 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:22.603250 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:22.608153 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:22.608428 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:22.609383 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:22.611135 16951 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:56:22.615835 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:56:22.615981 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 00:56:22.616076 16889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:56:22.619292 16889 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:56:22.619395 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:22.619831 16951 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Bootstrap starting.
I20260927 00:56:22.633590 16889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41555
I20260927 00:56:22.634097 17058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41555 every 8 connection(s)
I20260927 00:56:22.635332 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:22.636749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 16889
I20260927 00:56:22.637255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42611
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37497
--webserver_port=43473
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:56:22.641983 16951 log.cc:824] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:22.651862 16951 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:56:22.652329 16951 tablet_bootstrap.cc:492] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Bootstrap complete.
I20260927 00:56:22.653661 16951 ts_tablet_manager.cc:1397] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.016s
I20260927 00:56:22.657018 16951 raft_consensus.cc:348] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:22.657655 16951 raft_consensus.cc:729] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00081122823f4bee9b08b4a7636221f6, State: Initialized, Role: FOLLOWER
I20260927 00:56:22.658237 16951 consensus_queue.cc:260] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:22.659466 16951 ts_tablet_manager.cc:1428] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260927 00:56:22.701864 17061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:56:22.638299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:55235)
0927 00:56:22.639041 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:22.639108 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:22.640401 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:22.700700 (+ 60299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":227,"threads_started":1}
W20260927 00:56:22.702584 17059 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37497 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37497: Client connection negotiation failed: client connection to 127.0.98.62:37497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:56:22.703539 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:22.703116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:55225)
0927 00:56:22.703341 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:22.703451 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 00:56:22.704487 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:22.704149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:44803)
0927 00:56:22.704331 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:22.704397 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 00:56:22.704792 17059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:56:22.774816 17062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:22.775262 17062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:22.775349 17062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:22.780153 17062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:22.780256 17062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:22.780354 17062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:22.786620 17062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43473
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:22.788226 17062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:22.789593 17062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:22.801532 17069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:22.801543 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:22.801759 17070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:22.801802 17062 server_base.cc:1034] running on GCE node
I20260927 00:56:22.802454 17062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:22.803299 17062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:22.804499 17062 hybrid_clock.cc:648] HybridClock initialized: now 1790470582804461 us; error 54 us; skew 500 ppm
Sep 27 00:56:22 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:22.807860 17062 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:56:22.809100 17062 webserver.cc:460] Webserver started at http://127.0.98.2:43473/ using document root <none> and password file <none>
I20260927 00:56:22.809798 17062 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:22.809867 17062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:22.814918 17062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 00:56:22.817865 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:22.819636 17062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:22.819813 17062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "660ddab072ac486fba80c4d4bc18bad7"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "350b83688314d29f6076284fd874e111"
server_key_iv: "e9f8ef2d42ec767947a3d63e1027dbd0"
server_key_version: "encryptionkey@0"
I20260927 00:56:22.820568 17062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:22.839689 17062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:22.843659 17062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:22.843895 17062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:22.844635 17062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:22.845978 17086 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 00:56:22.850062 17086 ts_tablet_manager.cc:1852] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 00:56:22.850489 17086 ts_tablet_manager.cc:1905] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:22.854835 17086 ts_tablet_manager.cc:1918] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:22.854950 17086 log.cc:1196] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:22.855315 17086 ts_tablet_manager.cc:1939] T 4bede60cedcb47a89519b105c7b41d1a P 660ddab072ac486fba80c4d4bc18bad7: Deleting consensus metadata
I20260927 00:56:22.855651 17062 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 00:56:22.855775 17062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.001s
I20260927 00:56:22.855883 17062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:22.855913 17062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:22.867270 17062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42611
I20260927 00:56:22.867313 17193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42611 every 8 connection(s)
I20260927 00:56:22.868426 17062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:22.872678 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:22.871086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:53095)
0927 00:56:22.871705 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:22.871843 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":136,"threads_started":1}
W20260927 00:56:22.873270 17194 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37497 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37497: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
I20260927 00:56:22.874013 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:22.873784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:56979)
0927 00:56:22.873908 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:22.873957 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:56:22.874652 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:22.874456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:57513)
0927 00:56:22.874552 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:22.874595 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:56:22.874846 17194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:56:22.878070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17062
I20260927 00:56:22.878545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35077
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37497
--webserver_port=42521
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:23.011449 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:23.011976 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:23.012079 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:23.017035 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:23.017148 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:23.017220 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:23.023743 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42521
--tserver_master_addrs=127.0.98.62:37497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:23.025290 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:23.026687 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:23.038358 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:23.038480 17197 server_base.cc:1034] running on GCE node
W20260927 00:56:23.038390 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:23.038383 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:23.039103 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:23.040005 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:23.041209 17197 hybrid_clock.cc:648] HybridClock initialized: now 1790470583041169 us; error 58 us; skew 500 ppm
Sep 27 00:56:23 dist-test-slave-9fh2 krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:23.044684 17197 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:56:23.045938 17197 webserver.cc:460] Webserver started at http://127.0.98.3:42521/ using document root <none> and password file <none>
I20260927 00:56:23.046610 17197 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:23.046676 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:23.051733 17197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:56:23.054667 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:23.056473 17197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:23.056639 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "3cbc4dae5160440d82e2dd96f62a095c"
format_stamp: "Formatted at 2026-09-27 00:55:40 on dist-test-slave-9fh2"
server_key: "42db248746cb81798f867634983bb682"
server_key_iv: "763e0eb989afc9aed9ea0cdb4eefdb44"
server_key_version: "encryptionkey@0"
I20260927 00:56:23.057396 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:23.080377 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:23.084209 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:23.084439 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:23.085201 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:23.086304 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:23.086364 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:23.086416 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:23.086447 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:23.097775 17197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35077
I20260927 00:56:23.097829 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35077 every 8 connection(s)
I20260927 00:56:23.099126 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:23.103983 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:23.102062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:33843)
0927 00:56:23.102965 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:23.103171 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":160,"threads_started":1}
W20260927 00:56:23.104682 17326 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37497 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37497: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
I20260927 00:56:23.105577 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:23.105339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:58981)
0927 00:56:23.105463 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:23.105516 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 00:56:23.106262 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:23.106066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:39987)
0927 00:56:23.106171 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:23.106212 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:56:23.106420 17326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:56:23.107079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17197
I20260927 00:56:23.706036 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:23.705527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:36109)
0927 00:56:23.705873 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:23.705935 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260927 00:56:23.876003 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:23.875427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:56219)
0927 00:56:23.875783 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:23.875873 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260927 00:56:24.107683 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.107133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:46983)
0927 00:56:24.107470 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:24.107534 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260927 00:56:24.439591 17333 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:24.439810 17333 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:24.441010 17333 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
Sep 27 00:56:24 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:56:24 dist-test-slave-9fh2 krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:56:24.457402 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.441552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54139 (local address 127.0.98.3:35077)
0927 00:56:24.442413 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:24.442437 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:24.442471 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:56:24.443430 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:24.443438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:24.443886 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:24.444240 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:24.445681 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.446826 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.448168 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.448675 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.448850 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:24.452189 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:24.452199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:24.452223 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:24.452265 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:24.454510 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:24.455236 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:24.455241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:24.455246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:24.455386 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:24.455816 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:24.455819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:24.455821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:24.456287 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:24.456415 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:24.456934 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 00:56:24.457228 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":130,"threads_started":1}
I20260927 00:56:24.457778 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.441755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53019 (local address 127.0.98.2:42611)
0927 00:56:24.442270 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:24.442290 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:24.442348 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:56:24.443347 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:24.443355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:24.443763 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:24.444049 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:24.445666 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.446716 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.448171 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.448660 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.448804 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:24.452313 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:24.452318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:24.452331 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:24.452363 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:24.454504 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:24.455589 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:24.455596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:24.455601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:24.455851 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:24.456164 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:24.456166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:24.456168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:24.456677 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:24.456830 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:24.457205 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 00:56:24.457580 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":109,"threads_started":1}
I20260927 00:56:24.457399 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.441397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35077 (local address 127.0.98.1:54139)
0927 00:56:24.441953 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:24.441971 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:24.442110 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:24.444223 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:24.444230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:24.444869 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:24.445466 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:24.445488 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.446991 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.447001 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:24.448040 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:24.448050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.448819 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.448823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:24.449057 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:24.449723 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:24.449746 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:24.451864 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:24.454674 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:24.454682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:24.454706 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:24.455047 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:24.455506 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:24.455511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:24.455514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:24.455714 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:24.456404 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:24.456412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:24.456642 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 00:56:24.456806 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 00:56:24.457171 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":215,"spinlock_wait_cycles":6144,"thread_start_us":103,"threads_started":1}
I20260927 00:56:24.457590 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.441397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.98.1:53019)
0927 00:56:24.442223 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:24.442239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:24.442371 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:24.444237 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:24.444241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:24.444857 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:24.445465 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:24.445486 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.446871 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.446875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:24.448035 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:24.448046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:24.448796 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:24.448799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:24.449055 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:24.449871 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:24.449884 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:24.452072 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:24.455219 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:24.455222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:24.455224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:24.455430 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:24.455966 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:24.455970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:24.455972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:24.456063 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:24.456802 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:24.456805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:24.456922 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 00:56:24.457017 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:56:24.457285 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":297,"spinlock_wait_cycles":23424,"thread_start_us":101,"threads_started":1}
W20260927 00:56:24.461244 16948 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
W20260927 00:56:24.461468 16947 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:24.461542 16947 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6; no voters: 3cbc4dae5160440d82e2dd96f62a095c, 660ddab072ac486fba80c4d4bc18bad7
I20260927 00:56:24.461838 17333 raft_consensus.cc:2726] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:56:24.707366 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.706981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:40077)
0927 00:56:24.707190 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:24.707276 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:56:24.877533 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:24.876939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:51573)
0927 00:56:24.877275 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:24.877368 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
I20260927 00:56:25.109227 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:25.108625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:39115)
0927 00:56:25.108968 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:25.109059 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260927 00:56:25.708935 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:25.708233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:55463)
0927 00:56:25.708637 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:25.708730 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":87,"threads_started":1}
I20260927 00:56:25.879320 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:25.878593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:42271)
0927 00:56:25.878981 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:25.879067 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":97,"threads_started":1}
I20260927 00:56:26.110924 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:26.110185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:51737)
0927 00:56:26.110612 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:26.110707 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":84,"threads_started":1}
I20260927 00:56:26.502806 17343 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:26.503001 17343 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:26.503564 17343 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
W20260927 00:56:26.504496 16947 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
W20260927 00:56:26.504657 16948 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:26.504904 16948 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6; no voters: 3cbc4dae5160440d82e2dd96f62a095c, 660ddab072ac486fba80c4d4bc18bad7
I20260927 00:56:26.505084 17343 raft_consensus.cc:2726] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:56:26.710619 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:26.709998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:45221)
0927 00:56:26.710366 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:26.710485 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260927 00:56:26.880899 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:26.880245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:45457)
0927 00:56:26.880713 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:26.880771 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":132,"threads_started":1}
I20260927 00:56:27.112735 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:27.112115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:55197)
0927 00:56:27.112528 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:27.112622 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
I20260927 00:56:27.712576 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:27.711951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:53431)
0927 00:56:27.712378 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:27.712472 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":104,"threads_started":1}
I20260927 00:56:27.882575 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:27.881982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:38305)
0927 00:56:27.882339 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:27.882415 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260927 00:56:28.114403 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:28.113777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:40123)
0927 00:56:28.114220 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:28.114300 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":186,"threads_started":1}
I20260927 00:56:28.714069 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:28.713543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:44671)
0927 00:56:28.713889 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:28.713965 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260927 00:56:28.884059 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:28.883401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:57109)
0927 00:56:28.883766 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:28.883861 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
I20260927 00:56:28.907280 17352 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:28.907470 17352 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:28.908030 17352 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
W20260927 00:56:28.909190 16948 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
W20260927 00:56:28.909338 16947 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:28.909468 16947 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6; no voters: 3cbc4dae5160440d82e2dd96f62a095c, 660ddab072ac486fba80c4d4bc18bad7
I20260927 00:56:28.909667 17352 raft_consensus.cc:2726] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:56:29.115914 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:29.115324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:60249)
0927 00:56:29.115724 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:29.115803 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":96,"threads_started":1}
I20260927 00:56:29.715641 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:29.715073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:43281)
0927 00:56:29.715432 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:29.715507 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
I20260927 00:56:29.885870 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:29.885154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:47489)
0927 00:56:29.885658 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:29.885755 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":133,"threads_started":1}
I20260927 00:56:30.117410 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:30.116825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:42823)
0927 00:56:30.117204 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:30.117287 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":93,"threads_started":1}
I20260927 00:56:30.717185 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:30.716627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:58669)
0927 00:56:30.716973 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:30.717053 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260927 00:56:30.887382 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:30.886837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:46123)
0927 00:56:30.887162 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:30.887242 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260927 00:56:31.118903 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:31.118339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:60757)
0927 00:56:31.118714 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:31.118778 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":181,"threads_started":1}
I20260927 00:56:31.429080 17360 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:31.429243 17360 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:31.429742 17360 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
W20260927 00:56:31.430727 16948 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
W20260927 00:56:31.430928 16947 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:31.431020 16947 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6; no voters: 3cbc4dae5160440d82e2dd96f62a095c, 660ddab072ac486fba80c4d4bc18bad7
I20260927 00:56:31.431196 17360 raft_consensus.cc:2726] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:56:31.718727 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:31.718172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:46653)
0927 00:56:31.718505 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:31.718596 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260927 00:56:31.889075 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:31.888409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:44557)
0927 00:56:31.888847 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:31.888936 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260927 00:56:32.120478 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:32.119914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:50919)
0927 00:56:32.120248 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:32.120342 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260927 00:56:32.720537 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:32.719972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:54273)
0927 00:56:32.720308 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:32.720399 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260927 00:56:32.890638 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:32.889973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:53501)
0927 00:56:32.890368 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:32.890471 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":103,"threads_started":1}
I20260927 00:56:33.122131 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:33.121470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:41779)
0927 00:56:33.121896 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:33.121992 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":78,"threads_started":1}
I20260927 00:56:33.722157 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:33.721620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:43425)
0927 00:56:33.721976 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:33.722055 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260927 00:56:33.892228 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:33.891702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:60107)
0927 00:56:33.892054 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:33.892113 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":153,"threads_started":1}
I20260927 00:56:34.124095 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:34.123383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:53337)
0927 00:56:34.123797 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:34.123963 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
I20260927 00:56:34.723649 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:34.723128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:59037)
0927 00:56:34.723453 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:34.723519 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":86,"threads_started":1}
I20260927 00:56:34.893446 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:34.892925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:52195)
0927 00:56:34.893235 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:34.893313 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":73,"threads_started":1}
I20260927 00:56:35.125669 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:35.125078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:37561)
0927 00:56:35.125396 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:35.125531 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260927 00:56:35.725432 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:35.724728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:45307)
0927 00:56:35.725202 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:35.725301 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":102,"threads_started":1}
I20260927 00:56:35.894814 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:35.894233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:60281)
0927 00:56:35.894590 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:35.894670 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":88,"threads_started":1}
I20260927 00:56:36.127251 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:36.126645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:44673)
0927 00:56:36.127002 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:36.127084 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260927 00:56:36.726965 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:36.726443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:36643)
0927 00:56:36.726787 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:36.726864 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260927 00:56:36.896302 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:36.895774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:51701)
0927 00:56:36.896121 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:36.896193 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260927 00:56:37.128665 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:37.128148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.3:59903)
0927 00:56:37.128485 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:37.128558 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260927 00:56:37.520525 17379 raft_consensus.cc:482] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:37.520759 17379 raft_consensus.cc:504] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "660ddab072ac486fba80c4d4bc18bad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "3cbc4dae5160440d82e2dd96f62a095c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35077 } } peers { permanent_uuid: "00081122823f4bee9b08b4a7636221f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41555 } }
I20260927 00:56:37.521626 17379 leader_election.cc:288] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611), 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077)
W20260927 00:56:37.522456 16948 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cbc4dae5160440d82e2dd96f62a095c (127.0.98.3:35077): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
W20260927 00:56:37.522665 16947 leader_election.cc:341] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 660ddab072ac486fba80c4d4bc18bad7 (127.0.98.2:42611): Not found: Tablet not found: 4bede60cedcb47a89519b105c7b41d1a
I20260927 00:56:37.522734 16947 leader_election.cc:302] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00081122823f4bee9b08b4a7636221f6; no voters: 3cbc4dae5160440d82e2dd96f62a095c, 660ddab072ac486fba80c4d4bc18bad7
I20260927 00:56:37.522888 17379 raft_consensus.cc:2726] T 4bede60cedcb47a89519b105c7b41d1a P 00081122823f4bee9b08b4a7636221f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:56:37.728636 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:37.728030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.1:35473)
0927 00:56:37.728404 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:37.728492 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":73,"threads_started":1}
I20260927 00:56:37.897747 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:37.897295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37497 (local address 127.0.98.2:43613)
0927 00:56:37.897585 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:37.897641 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260927 00:56:38.110091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 16889
I20260927 00:56:38.116852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17062
I20260927 00:56:38.123159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17197
2026-09-27T00:56:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59441 ms)
[----------] 30 tests from SecurityITest (250102 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:38 dist-test-slave-9fh2 krb5kdc[17384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:38 dist-test-slave-9fh2 krb5kdc[17384](info): set up 2 sockets
Sep 27 00:56:38 dist-test-slave-9fh2 krb5kdc[17384](info): commencing operation
krb5kdc: starting...
W20260927 00:56:39.087097 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:56:39.219336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:39.352721 17397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:39.353150 17397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:39.353215 17397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:39.357847 17397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:39.357947 17397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:39.357975 17397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:39.357996 17397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:39.358017 17397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:39.364271 17397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:39.365734 17397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:39.367015 17397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:39.377288 17402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:39.377288 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:39.377429 17405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:39.377743 17397 server_base.cc:1034] running on GCE node
I20260927 00:56:39.378571 17397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:39.380054 17397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:39.381248 17397 hybrid_clock.cc:648] HybridClock initialized: now 1790470599381224 us; error 45 us; skew 500 ppm
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:39.384584 17397 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:56:39.385697 17397 webserver.cc:460] Webserver started at http://127.0.98.62:34185/ using document root <none> and password file <none>
I20260927 00:56:39.386374 17397 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:39.386440 17397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:39.386641 17397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:39.389473 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "4c76f35ebbf74ac5a1137dd24cadb963"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "c5b3e41753eccaf968764dd7bbb7a426"
server_key_iv: "4c0943fd0ab8ed580dcfa384f734bda5"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.390151 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c76f35ebbf74ac5a1137dd24cadb963"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "c5b3e41753eccaf968764dd7bbb7a426"
server_key_iv: "4c0943fd0ab8ed580dcfa384f734bda5"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.394873 17397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:56:39.397984 17412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.399564 17397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:39.399688 17397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "4c76f35ebbf74ac5a1137dd24cadb963"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "c5b3e41753eccaf968764dd7bbb7a426"
server_key_iv: "4c0943fd0ab8ed580dcfa384f734bda5"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.399889 17397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:39.421298 17397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:39.425174 17397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:39.425400 17397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:39.434399 17397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38329
I20260927 00:56:39.434543 17464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38329 every 8 connection(s)
I20260927 00:56:39.435695 17397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:39.437891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17397
I20260927 00:56:39.438030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:39.438369 392 external_mini_cluster.cc:1428] Setting key ef99ce3d79c6e0d3425c67fd919d8e0c
I20260927 00:56:39.439172 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:39.447661 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963: Bootstrap starting.
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:39.451041 17465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:39.452052 17465 log.cc:824] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:39.454849 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963: No bootstrap required, opened a new log
I20260927 00:56:39.456389 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:39.440073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.0.98.62:38329)
0927 00:56:39.440683 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:39.440710 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:39.440779 (+ 69us) server_negotiation.cc:407] Connection header received
0927 00:56:39.442013 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:39.442045 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:39.442555 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:39.443001 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:39.444140 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.445760 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.447043 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.447663 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.447831 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:39.450960 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:39.450967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:39.450985 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:39.451024 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:39.453081 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:39.453678 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:39.453684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:39.453689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:39.453823 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:39.454137 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:39.454140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:39.454141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:39.454611 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:39.454763 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:39.455233 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 00:56:39.455578 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":193,"threads_started":1}
I20260927 00:56:39.459566 17465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38329 } }
I20260927 00:56:39.459846 17465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:39.459882 17465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c76f35ebbf74ac5a1137dd24cadb963, State: Initialized, Role: FOLLOWER
I20260927 00:56:39.460376 17465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38329 } }
I20260927 00:56:39.460527 17465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:39.460630 17465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:39.460734 17465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:39.462090 17465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38329 } }
I20260927 00:56:39.462505 17465 leader_election.cc:302] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c76f35ebbf74ac5a1137dd24cadb963; no voters:
I20260927 00:56:39.462843 17465 leader_election.cc:288] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:39.463023 17470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:39.463317 17470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [term 1 LEADER]: Becoming Leader. State: Replica: 4c76f35ebbf74ac5a1137dd24cadb963, State: Running, Role: LEADER
I20260927 00:56:39.463795 17470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38329 } }
I20260927 00:56:39.464344 17465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:39.465695 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c76f35ebbf74ac5a1137dd24cadb963. Latest consensus state: current_term: 1 leader_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38329 } } }
I20260927 00:56:39.465698 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76f35ebbf74ac5a1137dd24cadb963" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38329 } } }
I20260927 00:56:39.465889 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:39.465889 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:39.466311 17479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:39.469574 17479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:39.475036 17479 catalog_manager.cc:1347] Generated new cluster ID: 1e8af923523a435b90abcfa31843b009
I20260927 00:56:39.475092 17479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:39.481585 17479 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:39.482470 17479 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:39.488773 17479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963: Generated new TSK 0
I20260927 00:56:39.489863 17479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:56:39.521605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38329
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:39.642706 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:39.643016 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:39.643095 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:39.647289 17491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:39.647351 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:39.647411 17491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:39.652778 17491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:39.653841 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:39.654832 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:39.665688 17496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:39.665688 17499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:39.666906 17497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:39.667761 17491 server_base.cc:1034] running on GCE node
I20260927 00:56:39.668344 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:39.669198 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:39.670400 17491 hybrid_clock.cc:648] HybridClock initialized: now 1790470599670345 us; error 68 us; skew 500 ppm
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:39.673559 17491 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:56:39.674578 17491 webserver.cc:460] Webserver started at http://127.0.98.1:43463/ using document root <none> and password file <none>
I20260927 00:56:39.675239 17491 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:39.675316 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:39.675525 17491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:39.677912 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa6cc82eea4c42dba0acf45d35a7dced"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "2697ac29d9ac4241f984d585d1651a20"
server_key_iv: "71f470f5bb5adb79a040ac2d3e19b8ff"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.678534 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa6cc82eea4c42dba0acf45d35a7dced"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "2697ac29d9ac4241f984d585d1651a20"
server_key_iv: "71f470f5bb5adb79a040ac2d3e19b8ff"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.682798 17491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:56:39.685859 17506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.687511 17491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:39.687659 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "fa6cc82eea4c42dba0acf45d35a7dced"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "2697ac29d9ac4241f984d585d1651a20"
server_key_iv: "71f470f5bb5adb79a040ac2d3e19b8ff"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.687770 17491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:39.707260 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:39.710233 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:39.710412 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:39.710984 17491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:39.711869 17491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:39.711917 17491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.711969 17491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:39.711997 17491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.721877 17491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39583
I20260927 00:56:39.721966 17619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39583 every 8 connection(s)
I20260927 00:56:39.722843 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:39.728334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17491
I20260927 00:56:39.728464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:56:39.728785 392 external_mini_cluster.cc:1428] Setting key 0cbd8603f386686bd3aeffaffb4f300a
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:56:39.738775 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:39.725191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46237 (local address 127.0.98.62:38329)
0927 00:56:39.725446 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:39.725451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:39.726330 (+ 879us) server_negotiation.cc:407] Connection header received
0927 00:56:39.727870 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:39.727874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:39.727940 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:39.728021 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:39.729567 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.730296 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.731610 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.731861 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.731954 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:39.734675 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:39.734679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:39.734681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:39.734708 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:39.736399 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:39.736957 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:39.736961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:39.736963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:39.737043 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:39.737425 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:39.737428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:39.737430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:39.737657 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:39.737745 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:39.738406 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 00:56:39.738575 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 00:56:39.740046 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:39.725529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38329 (local address 127.0.98.1:46237)
0927 00:56:39.726107 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:39.726162 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:39.727622 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:39.728241 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:39.728249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:39.728759 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:39.729399 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:39.729416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.730435 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.730438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:39.731481 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:39.731492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.732023 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.732027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:39.732205 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:39.732866 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:39.732885 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:39.734484 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:39.736533 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:39.736541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:39.736562 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:39.736861 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:39.737137 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:39.737140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:39.737142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:39.737337 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:39.737759 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:39.737767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:39.738085 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 00:56:39.738285 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 00:56:39.738715 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:56:39.741863 17620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38329
I20260927 00:56:39.742201 17620 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:39.742954 17620 heartbeater.cc:511] Master 127.0.98.62:38329 requested a full tablet report, sending...
I20260927 00:56:39.745234 17429 ts_manager.cc:194] Registered new tserver with Master: fa6cc82eea4c42dba0acf45d35a7dced (127.0.98.1:39583)
I20260927 00:56:39.746984 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:56:39.758934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38329
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:39.867442 17625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:39.867712 17625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:39.867770 17625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:39.871779 17625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:39.871846 17625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:39.871930 17625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:39.877194 17625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:39.878221 17625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:39.879205 17625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:39.888968 17631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:39.888968 17633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:39.889021 17630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:39.889436 17625 server_base.cc:1034] running on GCE node
I20260927 00:56:39.889849 17625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:39.890465 17625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:39.891642 17625 hybrid_clock.cc:648] HybridClock initialized: now 1790470599891599 us; error 67 us; skew 500 ppm
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:39.894578 17625 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:56:39.895570 17625 webserver.cc:460] Webserver started at http://127.0.98.2:35145/ using document root <none> and password file <none>
I20260927 00:56:39.896142 17625 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:39.896205 17625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:39.896391 17625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:39.898403 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac3774762dda4ef59ee21a8ee5689583"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "cf79ce9b892451e027aae9464bc0e4de"
server_key_iv: "f477e8bb431366341fb9e562fccff04e"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.898927 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac3774762dda4ef59ee21a8ee5689583"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "cf79ce9b892451e027aae9464bc0e4de"
server_key_iv: "f477e8bb431366341fb9e562fccff04e"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.902817 17625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:56:39.905767 17640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.907240 17625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:39.907361 17625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "ac3774762dda4ef59ee21a8ee5689583"
format_stamp: "Formatted at 2026-09-27 00:56:39 on dist-test-slave-9fh2"
server_key: "cf79ce9b892451e027aae9464bc0e4de"
server_key_iv: "f477e8bb431366341fb9e562fccff04e"
server_key_version: "encryptionkey@0"
I20260927 00:56:39.907475 17625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:39.930024 17625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:39.933369 17625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:39.933526 17625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:39.934116 17625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:39.934945 17625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:39.934994 17625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.935058 17625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:39.935082 17625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:39.944967 17625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32791
I20260927 00:56:39.945021 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32791 every 8 connection(s)
I20260927 00:56:39.945890 17625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:39.946151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17625
I20260927 00:56:39.946277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:56:39.946610 392 external_mini_cluster.cc:1428] Setting key e553e4b1a30e7bca0d80c36c61eacef4
Sep 27 00:56:39 dist-test-slave-9fh2 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:56:39.960525 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:39.948287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56425 (local address 127.0.98.62:38329)
0927 00:56:39.948494 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:39.948500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:39.949376 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:56:39.950637 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:39.950641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:39.950701 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:39.950771 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:39.951952 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.952481 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.953669 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.953893 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.953975 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:39.956485 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:39.956491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:39.956495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:39.956541 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:39.957966 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:39.958598 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:39.958601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:39.958602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:39.958686 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:39.959047 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:39.959050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:39.959052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:39.959395 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:39.959512 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:39.960218 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 00:56:39.960367 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:56:39.961493 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:39.948718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38329 (local address 127.0.98.2:56425)
0927 00:56:39.949202 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:39.949253 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:39.950431 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:39.950948 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:39.950954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:39.951358 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:39.951821 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:39.951833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.952601 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.952604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:39.953563 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:39.953573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:39.954034 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:39.954038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:39.954187 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:39.954753 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:39.954770 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:39.956248 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:39.958182 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:39.958188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:39.958203 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:39.958489 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:39.958779 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:39.958782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:39.958784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:39.958921 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:39.959548 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:39.959556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:39.959881 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 00:56:39.960140 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 00:56:39.960484 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260927 00:56:39.962966 17754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38329
I20260927 00:56:39.963403 17754 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:39.964216 17754 heartbeater.cc:511] Master 127.0.98.62:38329 requested a full tablet report, sending...
I20260927 00:56:39.965745 17429 ts_manager.cc:194] Registered new tserver with Master: ac3774762dda4ef59ee21a8ee5689583 (127.0.98.2:32791)
I20260927 00:56:39.966658 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:56:39.978353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38329
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:40.110234 17759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:40.110610 17759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:40.110663 17759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:40.115087 17759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:40.115211 17759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:40.115280 17759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:40.121280 17759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:40.122679 17759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:40.123903 17759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:40.135021 17767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:40.135033 17765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:40.135107 17764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:40.135576 17759 server_base.cc:1034] running on GCE node
I20260927 00:56:40.136097 17759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:40.136852 17759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:40.138036 17759 hybrid_clock.cc:648] HybridClock initialized: now 1790470600138012 us; error 41 us; skew 500 ppm
Sep 27 00:56:40 dist-test-slave-9fh2 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:40.141088 17759 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:56:40.142092 17759 webserver.cc:460] Webserver started at http://127.0.98.3:37741/ using document root <none> and password file <none>
I20260927 00:56:40.142678 17759 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:40.142745 17759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:40.142936 17759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:40.145143 17759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "324c57dfe78270cc4d0d422b1b620de0"
server_key_iv: "a600e14b03b4d9b17c867114f0a87503"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.145709 17759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "324c57dfe78270cc4d0d422b1b620de0"
server_key_iv: "a600e14b03b4d9b17c867114f0a87503"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.149945 17759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:40.152987 17774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.154464 17759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:56:40.154663 17759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "324c57dfe78270cc4d0d422b1b620de0"
server_key_iv: "a600e14b03b4d9b17c867114f0a87503"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.154780 17759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:40.170240 17759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:40.173435 17759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:40.173636 17759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:40.174273 17759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:40.175230 17759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:40.175282 17759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.175338 17759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:40.175364 17759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.185645 17759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40323
I20260927 00:56:40.185674 17887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40323 every 8 connection(s)
I20260927 00:56:40.186622 17759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:40.187294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17759
I20260927 00:56:40.187417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:56:40.187700 392 external_mini_cluster.cc:1428] Setting key 18667df5cda85ae667276801314827ca
Sep 27 00:56:40 dist-test-slave-9fh2 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:40.203377 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.189198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56013 (local address 127.0.98.62:38329)
0927 00:56:40.189327 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.189330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.190591 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 00:56:40.192168 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.192171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.192246 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.192347 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:40.193703 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.194427 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.195821 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.196106 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.196222 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.199054 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:40.199058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:40.199060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:40.199099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:40.200768 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:40.201516 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:40.201520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:40.201522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:40.201638 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:40.202032 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:40.202035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:40.202037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:40.202313 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:40.202424 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.203027 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.203216 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:56:40.204324 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.189743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38329 (local address 127.0.98.3:56013)
0927 00:56:40.190418 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.190472 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.191917 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.192512 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.192519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.193004 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:40.193525 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.193539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.194654 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.194658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.195664 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.195676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.196306 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.196309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.196477 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.197112 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:40.197129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:40.198793 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:40.201026 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:40.201033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:40.201049 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:40.201405 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:40.201743 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:40.201745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:40.201747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:40.201933 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:40.202424 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:40.202431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:40.202769 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.202957 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.203293 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":178,"threads_started":1}
I20260927 00:56:40.205778 17888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38329
I20260927 00:56:40.206164 17888 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:40.206869 17888 heartbeater.cc:511] Master 127.0.98.62:38329 requested a full tablet report, sending...
I20260927 00:56:40.208200 17429 ts_manager.cc:194] Registered new tserver with Master: 44747fa4cf6d4ab8af9a3d1b96da29b3 (127.0.98.3:40323)
I20260927 00:56:40.208835 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56013
I20260927 00:56:40.215227 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:40.225749 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.217376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.0.98.62:38329)
0927 00:56:40.217615 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.217622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.217913 (+ 291us) server_negotiation.cc:407] Connection header received
0927 00:56:40.218068 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.218071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.218134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.218231 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:40.219021 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.219684 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.220562 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.220799 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.220892 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.222240 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:40.222244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:40.222246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:40.222277 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:40.223916 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:40.224378 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:40.224382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:40.224383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:40.224470 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:40.224699 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:40.224701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:40.224702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:40.225022 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:40.225136 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.225393 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.225550 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:56:40.229262 17429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:40.232549 17429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:40.250599 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.244032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:38352)
0927 00:56:40.244809 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.244821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.244887 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.245646 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.245654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.245698 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:40.245895 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.245904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.247382 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.247384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.248983 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.248991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.249932 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.249935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.250059 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.250123 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.250227 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.250452 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":105,"spinlock_wait_cycles":6912,"thread_start_us":118,"threads_started":1}
I20260927 00:56:40.250609 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.245044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.0.98.3:40323)
0927 00:56:40.245535 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.245546 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.245571 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:56:40.245656 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.245661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.245884 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.246054 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:40.246410 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.247484 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.249053 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.249923 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.250030 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.250158 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.250296 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.250410 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":183,"threads_started":1}
I20260927 00:56:40.250960 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.244630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.0.98.1:39583)
0927 00:56:40.245210 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.245216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.245238 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:56:40.245311 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.245315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.245470 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.245617 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:40.246038 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.247237 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.249120 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.249799 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.250393 (+ 594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.250568 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.250628 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.250767 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":139,"threads_started":1}
I20260927 00:56:40.251341 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.244521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40323 (local address 127.0.0.1:60238)
0927 00:56:40.245041 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.245057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.245178 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.246204 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.246206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.246220 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:40.246298 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.246302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.247631 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.247634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.248864 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.248876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.250031 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.250034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.250100 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.250123 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.251044 (+ 921us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.251188 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":6,"thread_start_us":101,"threads_started":1}
I20260927 00:56:40.251816 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.243610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32791 (local address 127.0.0.1:57352)
0927 00:56:40.244611 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.244678 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.244888 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.245661 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.245664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.245701 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:40.245848 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.245854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.247376 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.247380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.248757 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.248768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.249880 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.249884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.250177 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.250210 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.250284 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.251692 (+ 1408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":230,"spinlock_wait_cycles":5760,"thread_start_us":114,"threads_started":1}
I20260927 00:56:40.251885 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.243799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.0.98.2:32791)
0927 00:56:40.244318 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.244331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.244770 (+ 439us) server_negotiation.cc:407] Connection header received
0927 00:56:40.245062 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.245078 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.245312 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.245443 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:40.245963 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.247237 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.248888 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.249730 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.249852 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.250033 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.251605 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.251722 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":138,"threads_started":1}
I20260927 00:56:40.254350 17822 tablet_service.cc:1467] Processing CreateTablet for tablet ef505f8ee2a04ae9b0c0dc856e02f0db (DEFAULT_TABLE table=test-table [id=10d9d32116b4423498f7ca19da27f668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:40.254819 17554 tablet_service.cc:1467] Processing CreateTablet for tablet ef505f8ee2a04ae9b0c0dc856e02f0db (DEFAULT_TABLE table=test-table [id=10d9d32116b4423498f7ca19da27f668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:40.254819 17688 tablet_service.cc:1467] Processing CreateTablet for tablet ef505f8ee2a04ae9b0c0dc856e02f0db (DEFAULT_TABLE table=test-table [id=10d9d32116b4423498f7ca19da27f668]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:40.255611 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef505f8ee2a04ae9b0c0dc856e02f0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:40.256090 17554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef505f8ee2a04ae9b0c0dc856e02f0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:40.256091 17688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef505f8ee2a04ae9b0c0dc856e02f0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:40.262764 17905 tablet_bootstrap.cc:492] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Bootstrap starting.
I20260927 00:56:40.264985 17906 tablet_bootstrap.cc:492] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced: Bootstrap starting.
I20260927 00:56:40.266067 17905 tablet_bootstrap.cc:654] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:40.266037 17907 tablet_bootstrap.cc:492] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: Bootstrap starting.
I20260927 00:56:40.267524 17905 log.cc:824] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:40.267838 17906 tablet_bootstrap.cc:654] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:40.268584 17906 log.cc:824] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:40.268978 17907 tablet_bootstrap.cc:654] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:40.269922 17907 log.cc:824] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:40.270295 17905 tablet_bootstrap.cc:492] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: No bootstrap required, opened a new log
I20260927 00:56:40.270475 17906 tablet_bootstrap.cc:492] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced: No bootstrap required, opened a new log
I20260927 00:56:40.270540 17905 ts_tablet_manager.cc:1397] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:56:40.270717 17906 ts_tablet_manager.cc:1397] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:56:40.272403 17907 tablet_bootstrap.cc:492] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: No bootstrap required, opened a new log
I20260927 00:56:40.272632 17907 ts_tablet_manager.cc:1397] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:56:40.274711 17906 raft_consensus.cc:348] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.274969 17906 raft_consensus.cc:374] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:40.275040 17906 raft_consensus.cc:729] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6cc82eea4c42dba0acf45d35a7dced, State: Initialized, Role: FOLLOWER
I20260927 00:56:40.275054 17905 raft_consensus.cc:348] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.275440 17905 raft_consensus.cc:374] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:40.275568 17905 raft_consensus.cc:729] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44747fa4cf6d4ab8af9a3d1b96da29b3, State: Initialized, Role: FOLLOWER
I20260927 00:56:40.275557 17906 consensus_queue.cc:260] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.276130 17905 consensus_queue.cc:260] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.276566 17906 ts_tablet_manager.cc:1428] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:56:40.276973 17620 heartbeater.cc:503] Master 127.0.98.62:38329 was elected leader, sending a full tablet report...
I20260927 00:56:40.277024 17888 heartbeater.cc:503] Master 127.0.98.62:38329 was elected leader, sending a full tablet report...
I20260927 00:56:40.276847 17907 raft_consensus.cc:348] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.277139 17907 raft_consensus.cc:374] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:40.277182 17907 raft_consensus.cc:729] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3774762dda4ef59ee21a8ee5689583, State: Initialized, Role: FOLLOWER
I20260927 00:56:40.277665 17907 consensus_queue.cc:260] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.278780 17905 ts_tablet_manager.cc:1428] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 00:56:40.278772 17907 ts_tablet_manager.cc:1428] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:40.279191 17754 heartbeater.cc:503] Master 127.0.98.62:38329 was elected leader, sending a full tablet report...
I20260927 00:56:40.301091 17913 raft_consensus.cc:482] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:40.301410 17913 raft_consensus.cc:504] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.302943 17913 leader_election.cc:288] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44747fa4cf6d4ab8af9a3d1b96da29b3 (127.0.98.3:40323), fa6cc82eea4c42dba0acf45d35a7dced (127.0.98.1:39583)
I20260927 00:56:40.307801 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.303331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.98.2:47309)
0927 00:56:40.303507 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.303525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.303630 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.304089 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.304094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.304117 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:40.304255 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.304273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.305291 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.305295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.306421 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.306430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.307254 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.307258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.307406 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.307438 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.307529 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.307630 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 00:56:40.307834 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.303475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47309 (local address 127.0.98.1:39583)
0927 00:56:40.303600 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.303605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.303626 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:56:40.303867 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.303869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.303923 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.304033 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:40.304422 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.305115 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.306551 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.307104 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.307247 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.307330 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.307573 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.307667 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:56:40.308110 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.303331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40323 (local address 127.0.98.2:54225)
0927 00:56:40.304082 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.304097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.304227 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.304548 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.304551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.304570 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:40.304729 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.304736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.305692 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.305695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.306766 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.306777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.307602 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.307606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.307762 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.307792 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.307876 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.307954 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":419,"thread_start_us":125,"threads_started":1}
I20260927 00:56:40.308115 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.303442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54225 (local address 127.0.98.3:40323)
0927 00:56:40.303600 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.303605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.304213 (+ 608us) server_negotiation.cc:407] Connection header received
0927 00:56:40.304321 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.304332 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.304392 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.304492 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:40.304869 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.305557 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.306909 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.307472 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.307571 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.307639 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.307910 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.307999 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:56:40.308720 17574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef505f8ee2a04ae9b0c0dc856e02f0db" candidate_uuid: "ac3774762dda4ef59ee21a8ee5689583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" is_pre_election: true
I20260927 00:56:40.308908 17842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef505f8ee2a04ae9b0c0dc856e02f0db" candidate_uuid: "ac3774762dda4ef59ee21a8ee5689583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" is_pre_election: true
I20260927 00:56:40.309132 17574 raft_consensus.cc:2454] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3774762dda4ef59ee21a8ee5689583 in term 0.
I20260927 00:56:40.309252 17842 raft_consensus.cc:2454] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3774762dda4ef59ee21a8ee5689583 in term 0.
I20260927 00:56:40.309736 17641 leader_election.cc:302] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac3774762dda4ef59ee21a8ee5689583, fa6cc82eea4c42dba0acf45d35a7dced; no voters:
I20260927 00:56:40.310055 17913 raft_consensus.cc:2781] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:40.310149 17913 raft_consensus.cc:482] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:40.310190 17913 raft_consensus.cc:3037] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:40.311576 17913 raft_consensus.cc:504] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.312042 17913 leader_election.cc:288] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [CANDIDATE]: Term 1 election: Requested vote from peers 44747fa4cf6d4ab8af9a3d1b96da29b3 (127.0.98.3:40323), fa6cc82eea4c42dba0acf45d35a7dced (127.0.98.1:39583)
I20260927 00:56:40.312568 17574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef505f8ee2a04ae9b0c0dc856e02f0db" candidate_uuid: "ac3774762dda4ef59ee21a8ee5689583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6cc82eea4c42dba0acf45d35a7dced"
I20260927 00:56:40.312543 17842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef505f8ee2a04ae9b0c0dc856e02f0db" candidate_uuid: "ac3774762dda4ef59ee21a8ee5689583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3"
I20260927 00:56:40.312764 17842 raft_consensus.cc:3037] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:40.312747 17574 raft_consensus.cc:3037] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:40.314174 17574 raft_consensus.cc:2454] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3774762dda4ef59ee21a8ee5689583 in term 1.
I20260927 00:56:40.314172 17842 raft_consensus.cc:2454] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3774762dda4ef59ee21a8ee5689583 in term 1.
I20260927 00:56:40.314548 17644 leader_election.cc:302] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44747fa4cf6d4ab8af9a3d1b96da29b3, ac3774762dda4ef59ee21a8ee5689583; no voters:
I20260927 00:56:40.314765 17913 raft_consensus.cc:2781] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:40.315006 17913 raft_consensus.cc:686] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 1 LEADER]: Becoming Leader. State: Replica: ac3774762dda4ef59ee21a8ee5689583, State: Running, Role: LEADER
I20260927 00:56:40.315419 17913 consensus_queue.cc:237] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } }
I20260927 00:56:40.318850 17429 catalog_manager.cc:5495] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3774762dda4ef59ee21a8ee5689583 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ac3774762dda4ef59ee21a8ee5689583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3774762dda4ef59ee21a8ee5689583" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } health_report { overall_health: UNKNOWN } } }
I20260927 00:56:40.331323 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.327400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.2:32791)
0927 00:56:40.327545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.327549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.327567 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:56:40.327663 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.327665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.327732 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.327838 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:40.328104 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.328670 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.329523 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.329752 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.329846 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.330155 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:40.330749 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:40.330863 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.330989 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.331054 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:56:40.338338 17574 raft_consensus.cc:1261] T ef505f8ee2a04ae9b0c0dc856e02f0db P fa6cc82eea4c42dba0acf45d35a7dced [term 1 FOLLOWER]: Refusing update from remote peer ac3774762dda4ef59ee21a8ee5689583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:40.338516 17842 raft_consensus.cc:1261] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 1 FOLLOWER]: Refusing update from remote peer ac3774762dda4ef59ee21a8ee5689583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:40.339282 17913 consensus_queue.cc:1035] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6cc82eea4c42dba0acf45d35a7dced" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:40.339519 17915 consensus_queue.cc:1035] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44747fa4cf6d4ab8af9a3d1b96da29b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:40.351120 17920 mvcc.cc:205] Tried to move back new op lower bound from 7333767578976350208 to 7333767578895085568. Current Snapshot: MvccSnapshot[applied={T|T < 7333767578976350208 or (T in {7333767578976350208})}]
I20260927 00:56:40.361706 17429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:40.361909 17429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:40.364691 17429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c76f35ebbf74ac5a1137dd24cadb963: Sending DeleteTablet for 3 replicas of tablet ef505f8ee2a04ae9b0c0dc856e02f0db
I20260927 00:56:40.365669 17554 tablet_service.cc:1514] Processing DeleteTablet for tablet ef505f8ee2a04ae9b0c0dc856e02f0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38352
I20260927 00:56:40.365883 17688 tablet_service.cc:1514] Processing DeleteTablet for tablet ef505f8ee2a04ae9b0c0dc856e02f0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57352
I20260927 00:56:40.366178 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet ef505f8ee2a04ae9b0c0dc856e02f0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60238
I20260927 00:56:40.366652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17491
I20260927 00:56:40.369714 17929 tablet_replica.cc:330] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: stopping tablet replica
I20260927 00:56:40.370213 17929 raft_consensus.cc:2229] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:56:40.370810 17929 raft_consensus.cc:2258] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:40.371371 17930 tablet_replica.cc:330] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: stopping tablet replica
I20260927 00:56:40.371771 17930 raft_consensus.cc:2229] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:40.372187 17930 raft_consensus.cc:2258] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:56:40.373670 17413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39583 (error 108)
I20260927 00:56:40.374248 17929 ts_tablet_manager.cc:1905] T ef505f8ee2a04ae9b0c0dc856e02f0db P ac3774762dda4ef59ee21a8ee5689583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:40.374701 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.374428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:38358)
0927 00:56:40.374543 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.374607 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 00:56:40.374815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17625
W20260927 00:56:40.374974 17413 catalog_manager.cc:4584] TS fa6cc82eea4c42dba0acf45d35a7dced (127.0.98.1:39583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef505f8ee2a04ae9b0c0dc856e02f0db: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
I20260927 00:56:40.376962 17930 ts_tablet_manager.cc:1905] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:40.379340 17930 ts_tablet_manager.cc:1918] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:40.379439 17930 log.cc:1196] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/ef505f8ee2a04ae9b0c0dc856e02f0db
I20260927 00:56:40.379699 17930 ts_tablet_manager.cc:1939] T ef505f8ee2a04ae9b0c0dc856e02f0db P 44747fa4cf6d4ab8af9a3d1b96da29b3: Deleting consensus metadata
I20260927 00:56:40.381003 17416 catalog_manager.cc:4841] TS 44747fa4cf6d4ab8af9a3d1b96da29b3 (127.0.98.3:40323): tablet ef505f8ee2a04ae9b0c0dc856e02f0db (table test-table [id=10d9d32116b4423498f7ca19da27f668]) successfully deleted
I20260927 00:56:40.382138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17759
I20260927 00:56:40.383028 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.382263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32791 (local address 127.0.0.1:57370)
0927 00:56:40.382852 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.382915 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
I20260927 00:56:40.388859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17397
2026-09-27T00:56:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2265 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T00:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:40Z Disabled control of system clock
I20260927 00:56:40.421303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37299
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:56:40.539279 17935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:40.539580 17935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:40.539618 17935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:40.544150 17935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:40.544230 17935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:40.544245 17935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:40.544263 17935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:40.544277 17935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:40.549325 17935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:40.550472 17935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:40.552111 17935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:40.561525 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:40.561525 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:40.561558 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:40.561844 17935 server_base.cc:1034] running on GCE node
I20260927 00:56:40.562455 17935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:40.563709 17935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:40.564877 17935 hybrid_clock.cc:648] HybridClock initialized: now 1790470600564858 us; error 44 us; skew 500 ppm
I20260927 00:56:40.566691 17935 webserver.cc:460] Webserver started at http://127.0.98.62:39101/ using document root <none> and password file <none>
I20260927 00:56:40.567340 17935 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:40.567411 17935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:40.567611 17935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:40.569694 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "271b588613f44d80af43e71abea7419a"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "0f2a65303ff15390e6169391e28bc0e9"
server_key_iv: "33baadba9b264708f7277dee7e0245fd"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.570220 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "271b588613f44d80af43e71abea7419a"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "0f2a65303ff15390e6169391e28bc0e9"
server_key_iv: "33baadba9b264708f7277dee7e0245fd"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.574028 17935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:56:40.576679 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.578121 17935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:56:40.578274 17935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "271b588613f44d80af43e71abea7419a"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "0f2a65303ff15390e6169391e28bc0e9"
server_key_iv: "33baadba9b264708f7277dee7e0245fd"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.578372 17935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:40.592585 17935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:40.593266 17935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:40.593436 17935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:40.600509 17935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37299
I20260927 00:56:40.600579 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37299 every 8 connection(s)
I20260927 00:56:40.601475 17935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:40.605590 18002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:40.607887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 17935
I20260927 00:56:40.608014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:40.608276 392 external_mini_cluster.cc:1428] Setting key 25004f1a15db79bacc3cb9bbc8a1eac3
I20260927 00:56:40.614249 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a: Bootstrap starting.
I20260927 00:56:40.617682 18002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:40.618680 18002 log.cc:824] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:40.620247 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.609904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.98.62:37299)
0927 00:56:40.610835 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.610863 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.610919 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:56:40.611855 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.611884 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.612525 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.613035 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:40.613577 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.615105 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.616080 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.616847 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.617042 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.618036 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:40.618046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:40.618067 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:40.618565 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:40.618641 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:40.618667 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:40.618777 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.618912 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.619304 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":135,"threads_started":1}
I20260927 00:56:40.621460 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a: No bootstrap required, opened a new log
I20260927 00:56:40.625901 18002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271b588613f44d80af43e71abea7419a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37299 } }
I20260927 00:56:40.626214 18002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:40.626297 18002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271b588613f44d80af43e71abea7419a, State: Initialized, Role: FOLLOWER
I20260927 00:56:40.626803 18002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271b588613f44d80af43e71abea7419a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37299 } }
I20260927 00:56:40.626947 18002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:40.626991 18002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:40.627075 18002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:40.628523 18002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271b588613f44d80af43e71abea7419a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37299 } }
I20260927 00:56:40.628923 18002 leader_election.cc:302] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 271b588613f44d80af43e71abea7419a; no voters:
I20260927 00:56:40.629294 18002 leader_election.cc:288] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:40.629482 18007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:40.629755 18007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [term 1 LEADER]: Becoming Leader. State: Replica: 271b588613f44d80af43e71abea7419a, State: Running, Role: LEADER
I20260927 00:56:40.630167 18007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271b588613f44d80af43e71abea7419a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37299 } }
I20260927 00:56:40.630802 18002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:40.632191 18008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "271b588613f44d80af43e71abea7419a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271b588613f44d80af43e71abea7419a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37299 } } }
I20260927 00:56:40.632355 18008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:40.632619 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 271b588613f44d80af43e71abea7419a. Latest consensus state: current_term: 1 leader_uuid: "271b588613f44d80af43e71abea7419a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271b588613f44d80af43e71abea7419a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37299 } } }
I20260927 00:56:40.633118 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:40.633150 18016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:40.637219 18016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:40.644162 18016 catalog_manager.cc:1347] Generated new cluster ID: af81b16f674b428a9051a39dce99ef2d
I20260927 00:56:40.644261 18016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:40.685664 18016 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:40.686949 18016 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:40.693543 18016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a: Generated new TSK 0
I20260927 00:56:40.694536 18016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:40.698529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37299
--builtin_ntp_servers=127.0.98.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:56:40.827344 18026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:40.827724 18026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:40.827785 18026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:40.832347 18026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:40.832466 18026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:40.832556 18026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:40.838127 18026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:40.839704 18026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:40.841740 18026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:40.853986 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:40.853971 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:40.853971 18032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:40.854570 18026 server_base.cc:1034] running on GCE node
I20260927 00:56:40.855085 18026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:40.855984 18026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:40.857174 18026 hybrid_clock.cc:648] HybridClock initialized: now 1790470600857158 us; error 33 us; skew 500 ppm
I20260927 00:56:40.859727 18026 webserver.cc:460] Webserver started at http://127.0.98.1:44973/ using document root <none> and password file <none>
I20260927 00:56:40.860477 18026 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:40.860558 18026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:40.860801 18026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:40.863474 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "59f54c61e1b64f93bf142a57d2995d74"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "ffc5b84ebab0c43a46bf3c4e94f4277b"
server_key_iv: "8bb96124b940425fe91b5809f67058b5"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.864147 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59f54c61e1b64f93bf142a57d2995d74"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "ffc5b84ebab0c43a46bf3c4e94f4277b"
server_key_iv: "8bb96124b940425fe91b5809f67058b5"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.869028 18026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:56:40.872176 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.873962 18026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:40.874162 18026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "59f54c61e1b64f93bf142a57d2995d74"
format_stamp: "Formatted at 2026-09-27 00:56:40 on dist-test-slave-9fh2"
server_key: "ffc5b84ebab0c43a46bf3c4e94f4277b"
server_key_iv: "8bb96124b940425fe91b5809f67058b5"
server_key_version: "encryptionkey@0"
I20260927 00:56:40.874329 18026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:40.887043 18026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:40.888023 18026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:40.888252 18026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:40.889030 18026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:40.890331 18026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:40.890389 18026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.890457 18026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:40.890489 18026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:40.902634 18026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32797
I20260927 00:56:40.902701 18153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32797 every 8 connection(s)
I20260927 00:56:40.903913 18026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:40.906391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18026
I20260927 00:56:40.906513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:56:40.906804 392 external_mini_cluster.cc:1428] Setting key d5ef9264909aee106c951664bede0d51
I20260927 00:56:40.909102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37299
--builtin_ntp_servers=127.0.98.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:56:40.916272 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.906272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51555 (local address 127.0.98.62:37299)
0927 00:56:40.906467 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:40.906473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:40.907944 (+ 1471us) server_negotiation.cc:407] Connection header received
0927 00:56:40.909962 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:40.909965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:40.910023 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:40.910111 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:40.911612 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.912398 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.913737 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.914084 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.914211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.915673 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:40.915677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:40.915680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:40.915698 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:40.915728 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:40.915739 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:40.915829 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:40.916024 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:56:40.916106 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:56:40.917366 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:40.906823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37299 (local address 127.0.98.1:51555)
0927 00:56:40.907740 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:40.907804 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:40.909644 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:40.910376 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:40.910387 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:40.910541 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:40.911442 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.911459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.912546 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.912549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.913583 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:40.913595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:40.914307 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:40.914311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:40.914501 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:40.915312 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:40.915314 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:40.915372 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:40.915458 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:40.915845 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:40.915856 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:40.915862 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:56:40.915981 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:56:40.916313 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":149,"threads_started":1}
I20260927 00:56:40.919096 18154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37299
I20260927 00:56:40.919440 18154 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:40.920114 18154 heartbeater.cc:511] Master 127.0.98.62:37299 requested a full tablet report, sending...
I20260927 00:56:40.921973 17964 ts_manager.cc:194] Registered new tserver with Master: 59f54c61e1b64f93bf142a57d2995d74 (127.0.98.1:32797)
I20260927 00:56:40.923420 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51555
W20260927 00:56:41.031021 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:41.031388 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:41.031431 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:41.036010 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:41.036092 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:41.036157 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:41.041107 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:41.042266 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:41.043708 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:41.054250 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:41.054271 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:41.054271 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:41.054519 18157 server_base.cc:1034] running on GCE node
I20260927 00:56:41.055124 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:41.055867 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:41.057077 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790470601057060 us; error 41 us; skew 500 ppm
I20260927 00:56:41.059315 18157 webserver.cc:460] Webserver started at http://127.0.98.2:41787/ using document root <none> and password file <none>
I20260927 00:56:41.059981 18157 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:41.060047 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:41.060246 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:41.062441 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "12120ce7e49744ae99d165b6d2c9ed0b"
format_stamp: "Formatted at 2026-09-27 00:56:41 on dist-test-slave-9fh2"
server_key: "e8dcb0a65082fda7cb34c954afae249f"
server_key_iv: "0c2a21ffaaec9b56d33f06860b7c2765"
server_key_version: "encryptionkey@0"
I20260927 00:56:41.063009 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12120ce7e49744ae99d165b6d2c9ed0b"
format_stamp: "Formatted at 2026-09-27 00:56:41 on dist-test-slave-9fh2"
server_key: "e8dcb0a65082fda7cb34c954afae249f"
server_key_iv: "0c2a21ffaaec9b56d33f06860b7c2765"
server_key_version: "encryptionkey@0"
I20260927 00:56:41.067476 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:56:41.070567 18171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:41.072307 18157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:41.072464 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "12120ce7e49744ae99d165b6d2c9ed0b"
format_stamp: "Formatted at 2026-09-27 00:56:41 on dist-test-slave-9fh2"
server_key: "e8dcb0a65082fda7cb34c954afae249f"
server_key_iv: "0c2a21ffaaec9b56d33f06860b7c2765"
server_key_version: "encryptionkey@0"
I20260927 00:56:41.072598 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:41.087924 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:41.088656 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:41.088835 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:41.089438 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:41.090435 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:41.090482 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:41.090538 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:41.090566 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:41.100417 18157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36843
I20260927 00:56:41.100486 18284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36843 every 8 connection(s)
I20260927 00:56:41.101379 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:41.107870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18157
I20260927 00:56:41.108049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:56:41.108371 392 external_mini_cluster.cc:1428] Setting key c2f69a8c7aa8d78de11ee37e85840eb5
I20260927 00:56:41.110713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37299
--builtin_ntp_servers=127.0.98.20:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:56:41.112723 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.103687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40867 (local address 127.0.98.62:37299)
0927 00:56:41.103889 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.103894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.105186 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 00:56:41.106821 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.106824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.106883 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.106976 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:41.108285 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.108981 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.110474 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.110711 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.110803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.112173 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:41.112177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:41.112180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:41.112199 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:41.112231 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:41.112240 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:41.112314 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.112532 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.112605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:56:41.114082 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.104088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37299 (local address 127.0.98.2:40867)
0927 00:56:41.104992 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.105047 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.106557 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.107209 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.107218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.107326 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:41.108094 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.108115 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.109173 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.109176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.110345 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.110355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.110873 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.110877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.111052 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.111774 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:41.111775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:41.111824 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:41.111911 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:41.112335 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:41.112344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:41.112351 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.112495 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.112888 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":113,"threads_started":1}
I20260927 00:56:41.115486 18285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37299
I20260927 00:56:41.115841 18285 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:41.116672 18285 heartbeater.cc:511] Master 127.0.98.62:37299 requested a full tablet report, sending...
I20260927 00:56:41.118029 17964 ts_manager.cc:194] Registered new tserver with Master: 12120ce7e49744ae99d165b6d2c9ed0b (127.0.98.2:36843)
I20260927 00:56:41.118643 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40867
W20260927 00:56:41.241832 18288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:41.242174 18288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:41.242216 18288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:41.247123 18288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:41.247247 18288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:41.247315 18288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:41.253135 18288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:41.254621 18288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:41.256480 18288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:41.268904 18296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:41.268882 18293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:41.268904 18294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:41.269160 18288 server_base.cc:1034] running on GCE node
I20260927 00:56:41.269932 18288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:41.270732 18288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:41.271912 18288 hybrid_clock.cc:648] HybridClock initialized: now 1790470601271893 us; error 35 us; skew 500 ppm
I20260927 00:56:41.274261 18288 webserver.cc:460] Webserver started at http://127.0.98.3:38693/ using document root <none> and password file <none>
I20260927 00:56:41.274976 18288 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:41.275053 18288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:41.275337 18288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:41.277930 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "736d2c575fce4b3ea4514132b22a6c14"
format_stamp: "Formatted at 2026-09-27 00:56:41 on dist-test-slave-9fh2"
server_key: "48f860a6ba3dc49b2940e35926f1cbcb"
server_key_iv: "abb9b1c06b4adf602fe19604a3db9c8c"
server_key_version: "encryptionkey@0"
I20260927 00:56:41.278630 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "736d2c575fce4b3ea4514132b22a6c14"
format_stamp: "Formatted at 2026-09-27 00:56:41 on dist-test-slave-9fh2"
server_key: "48f860a6ba3dc49b2940e35926f1cbcb"
server_key_iv: "abb9b1c06b4adf602fe19604a3db9c8c"
server_key_version: "encryptionkey@0"
I20260927 00:56:41.283573 18288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:56:41.286867 18302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:41.288686 18288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:41.288859 18288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "736d2c575fce4b3ea4514132b22a6c14"
format_stamp: "Formatted at 2026-09-27 00:56:41 on dist-test-slave-9fh2"
server_key: "48f860a6ba3dc49b2940e35926f1cbcb"
server_key_iv: "abb9b1c06b4adf602fe19604a3db9c8c"
server_key_version: "encryptionkey@0"
I20260927 00:56:41.289000 18288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:41.299788 18288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:41.300704 18288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:41.300936 18288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:41.301699 18288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:41.302927 18288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:41.302985 18288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:41.303046 18288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:41.303071 18288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:41.314199 18288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41275
I20260927 00:56:41.314229 18415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41275 every 8 connection(s)
I20260927 00:56:41.315331 18288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:41.319200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18288
I20260927 00:56:41.319342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:56:41.319684 392 external_mini_cluster.cc:1428] Setting key 62d24a8c9017eeb1036ac9730cdbe1e1
I20260927 00:56:41.325773 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.317812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60785 (local address 127.0.98.62:37299)
0927 00:56:41.317962 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.317966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.319475 (+ 1509us) server_negotiation.cc:407] Connection header received
0927 00:56:41.321094 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.321097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.321154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.321235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:41.322180 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.322707 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.323904 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.324119 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.324205 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.325310 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:41.325313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:41.325315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:41.325327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:41.325345 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:41.325352 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:41.325409 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.325553 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.325617 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:56:41.326623 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.318310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37299 (local address 127.0.98.3:60785)
0927 00:56:41.319280 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.319351 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.320877 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.321418 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.321425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.321516 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:41.322041 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.322054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.322826 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.322829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.323786 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.323796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.324268 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.324272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.324428 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.325044 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:41.325045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:41.325088 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:41.325157 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:41.325424 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:41.325433 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:41.325439 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.325547 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.325851 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":164,"threads_started":1}
I20260927 00:56:41.327952 18416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37299
I20260927 00:56:41.328253 18416 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:41.328930 18416 heartbeater.cc:511] Master 127.0.98.62:37299 requested a full tablet report, sending...
I20260927 00:56:41.330176 17964 ts_manager.cc:194] Registered new tserver with Master: 736d2c575fce4b3ea4514132b22a6c14 (127.0.98.3:41275)
I20260927 00:56:41.330847 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60785
I20260927 00:56:41.334613 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:41.340596 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.336601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.0.98.62:37299)
0927 00:56:41.336787 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.336792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.336931 (+ 139us) server_negotiation.cc:407] Connection header received
0927 00:56:41.337086 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.337089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.337138 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.337204 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:41.337523 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.338093 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.338978 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.339234 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.339334 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.340169 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:41.340173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:41.340175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:41.340187 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:41.340205 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:41.340213 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:41.340272 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.340360 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.340424 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:56:41.344004 17964 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:41.347779 17964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:41.367043 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.359335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41275 (local address 127.0.0.1:44366)
0927 00:56:41.360954 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.361026 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.361195 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.362264 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.362268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.362294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.362424 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.362431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.363681 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.363684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.365112 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.365123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.366466 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.366470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.366581 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.366626 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.366706 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.366863 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":8,"spinlock_wait_cycles":10624,"thread_start_us":107,"threads_started":1}
I20260927 00:56:41.367108 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.360709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.98.2:36843)
0927 00:56:41.361125 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.361136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.361163 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:56:41.361395 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.361400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.361626 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.361791 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.362523 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.363802 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.365348 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.366087 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.366194 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.366399 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.366830 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.366946 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":125,"threads_started":1}
I20260927 00:56:41.367097 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.360012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.0.98.1:32797)
0927 00:56:41.360402 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.360415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.361151 (+ 736us) server_negotiation.cc:407] Connection header received
0927 00:56:41.361394 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.361400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.361625 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.361791 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.362446 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.363830 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.365129 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.365943 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.366054 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.366327 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.366774 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.366926 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":174,"threads_started":1}
I20260927 00:56:41.367570 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.359501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32797 (local address 127.0.0.1:42870)
0927 00:56:41.360954 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.361026 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.361233 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.361898 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.361904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.361930 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.362090 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.362096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.363965 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.363968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.364994 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.365004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.366390 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.366393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.366534 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.366582 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.367317 (+ 735us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.367428 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":76,"spinlock_wait_cycles":897792,"thread_start_us":175,"threads_started":1}
I20260927 00:56:41.367660 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.359445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.0.98.3:41275)
0927 00:56:41.360685 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.360708 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.361237 (+ 529us) server_negotiation.cc:407] Connection header received
0927 00:56:41.361985 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.361989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.362128 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.362254 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.362552 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.363568 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.365256 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.366083 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.366194 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.366398 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.367396 (+ 998us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.367498 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":144,"threads_started":1}
I20260927 00:56:41.367803 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.360550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36843 (local address 127.0.0.1:37194)
0927 00:56:41.360954 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.361022 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.361194 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.362233 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.362236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.362253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.362366 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.362374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.363944 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.363947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.365209 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.365217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.366466 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.366470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.366644 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.366663 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.366720 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.367527 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":819,"spinlock_wait_cycles":61184,"thread_start_us":111,"threads_started":1}
I20260927 00:56:41.370949 18219 tablet_service.cc:1467] Processing CreateTablet for tablet 75658ce282204e778f26b25d7f46a574 (DEFAULT_TABLE table=test-table [id=42f9b81d96d04e74bd0ec138d740c6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:41.370949 18350 tablet_service.cc:1467] Processing CreateTablet for tablet 75658ce282204e778f26b25d7f46a574 (DEFAULT_TABLE table=test-table [id=42f9b81d96d04e74bd0ec138d740c6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:41.372255 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75658ce282204e778f26b25d7f46a574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:41.372268 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75658ce282204e778f26b25d7f46a574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:41.373040 18088 tablet_service.cc:1467] Processing CreateTablet for tablet 75658ce282204e778f26b25d7f46a574 (DEFAULT_TABLE table=test-table [id=42f9b81d96d04e74bd0ec138d740c6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:41.374210 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75658ce282204e778f26b25d7f46a574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:41.380517 18433 tablet_bootstrap.cc:492] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Bootstrap starting.
I20260927 00:56:41.381947 18435 tablet_bootstrap.cc:492] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: Bootstrap starting.
I20260927 00:56:41.382437 18434 tablet_bootstrap.cc:492] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: Bootstrap starting.
I20260927 00:56:41.384315 18435 tablet_bootstrap.cc:654] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:41.384897 18434 tablet_bootstrap.cc:654] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:41.384913 18433 tablet_bootstrap.cc:654] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:41.385069 18435 log.cc:824] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:41.385870 18433 log.cc:824] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:41.385862 18434 log.cc:824] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:41.387118 18435 tablet_bootstrap.cc:492] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: No bootstrap required, opened a new log
I20260927 00:56:41.387337 18435 ts_tablet_manager.cc:1397] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:41.388015 18433 tablet_bootstrap.cc:492] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: No bootstrap required, opened a new log
I20260927 00:56:41.388229 18433 ts_tablet_manager.cc:1397] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260927 00:56:41.388454 18434 tablet_bootstrap.cc:492] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: No bootstrap required, opened a new log
I20260927 00:56:41.388623 18434 ts_tablet_manager.cc:1397] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 00:56:41.391016 18435 raft_consensus.cc:348] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.391316 18435 raft_consensus.cc:374] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:41.391379 18435 raft_consensus.cc:729] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f54c61e1b64f93bf142a57d2995d74, State: Initialized, Role: FOLLOWER
I20260927 00:56:41.391812 18435 consensus_queue.cc:260] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.392613 18154 heartbeater.cc:503] Master 127.0.98.62:37299 was elected leader, sending a full tablet report...
I20260927 00:56:41.392856 18435 ts_tablet_manager.cc:1428] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:56:41.393263 18434 raft_consensus.cc:348] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.393568 18434 raft_consensus.cc:374] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:41.393616 18434 raft_consensus.cc:729] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12120ce7e49744ae99d165b6d2c9ed0b, State: Initialized, Role: FOLLOWER
I20260927 00:56:41.394142 18434 consensus_queue.cc:260] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.394958 18285 heartbeater.cc:503] Master 127.0.98.62:37299 was elected leader, sending a full tablet report...
I20260927 00:56:41.395427 18434 ts_tablet_manager.cc:1428] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260927 00:56:41.395766 18433 raft_consensus.cc:348] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.396147 18433 raft_consensus.cc:374] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:41.396209 18433 raft_consensus.cc:729] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736d2c575fce4b3ea4514132b22a6c14, State: Initialized, Role: FOLLOWER
I20260927 00:56:41.396699 18433 consensus_queue.cc:260] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.397854 18433 ts_tablet_manager.cc:1428] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 00:56:41.398259 18416 heartbeater.cc:503] Master 127.0.98.62:37299 was elected leader, sending a full tablet report...
W20260927 00:56:41.405319 18155 tablet.cc:2369] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:56:41.500793 18440 raft_consensus.cc:482] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:41.501022 18440 raft_consensus.cc:504] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.502267 18440 leader_election.cc:288] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736d2c575fce4b3ea4514132b22a6c14 (127.0.98.3:41275), 59f54c61e1b64f93bf142a57d2995d74 (127.0.98.1:32797)
I20260927 00:56:41.505506 18439 raft_consensus.cc:482] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:41.505816 18439 raft_consensus.cc:504] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.507038 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.502625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32797 (local address 127.0.98.2:43029)
0927 00:56:41.502797 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.502815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.502933 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.503339 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.503343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.503383 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.503551 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.503561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.504531 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.504535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.505617 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.505629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.506586 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.506590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.506701 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.506723 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.506794 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.506883 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":9}
I20260927 00:56:41.507047 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.502891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43029 (local address 127.0.98.1:32797)
0927 00:56:41.503026 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.503031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.503046 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:56:41.503120 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.503123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.503202 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.503290 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.503736 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.504417 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.505782 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.506429 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.506545 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.506628 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.506827 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.506938 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:56:41.507325 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.502869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54609 (local address 127.0.98.3:41275)
0927 00:56:41.503067 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.503073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.503354 (+ 281us) server_negotiation.cc:407] Connection header received
0927 00:56:41.503467 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.503470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.503543 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.503664 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.504067 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.504686 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.506016 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.506465 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.506573 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.506641 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.507035 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.507173 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 00:56:41.507580 18439 leader_election.cc:288] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736d2c575fce4b3ea4514132b22a6c14 (127.0.98.3:41275), 12120ce7e49744ae99d165b6d2c9ed0b (127.0.98.2:36843)
I20260927 00:56:41.507812 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.502704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41275 (local address 127.0.98.2:54609)
0927 00:56:41.503246 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.503261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.503358 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.503739 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.503742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.503772 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.503920 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.503928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.504811 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.504814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.505862 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.505874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.506740 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.506744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.506867 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.506893 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.507547 (+ 654us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.507648 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":26,"thread_start_us":152,"threads_started":1}
I20260927 00:56:41.507893 18108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75658ce282204e778f26b25d7f46a574" candidate_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f54c61e1b64f93bf142a57d2995d74" is_pre_election: true
I20260927 00:56:41.508154 18108 raft_consensus.cc:2454] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12120ce7e49744ae99d165b6d2c9ed0b in term 0.
I20260927 00:56:41.508507 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75658ce282204e778f26b25d7f46a574" candidate_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736d2c575fce4b3ea4514132b22a6c14" is_pre_election: true
I20260927 00:56:41.508821 18172 leader_election.cc:302] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12120ce7e49744ae99d165b6d2c9ed0b, 59f54c61e1b64f93bf142a57d2995d74; no voters:
I20260927 00:56:41.508859 18370 raft_consensus.cc:2454] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12120ce7e49744ae99d165b6d2c9ed0b in term 0.
I20260927 00:56:41.509250 18440 raft_consensus.cc:2781] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:41.509346 18440 raft_consensus.cc:482] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:41.509397 18440 raft_consensus.cc:3037] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:41.511004 18440 raft_consensus.cc:504] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.511467 18440 leader_election.cc:288] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [CANDIDATE]: Term 1 election: Requested vote from peers 736d2c575fce4b3ea4514132b22a6c14 (127.0.98.3:41275), 59f54c61e1b64f93bf142a57d2995d74 (127.0.98.1:32797)
I20260927 00:56:41.512034 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75658ce282204e778f26b25d7f46a574" candidate_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736d2c575fce4b3ea4514132b22a6c14"
I20260927 00:56:41.512039 18108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75658ce282204e778f26b25d7f46a574" candidate_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f54c61e1b64f93bf142a57d2995d74"
I20260927 00:56:41.512194 18108 raft_consensus.cc:3037] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:41.512195 18370 raft_consensus.cc:3037] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:41.512758 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.508211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54869 (local address 127.0.98.3:41275)
0927 00:56:41.508518 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.508522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.508786 (+ 264us) server_negotiation.cc:407] Connection header received
0927 00:56:41.509068 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.509071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.509121 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.509164 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.509646 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.510126 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.511319 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.511856 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.512000 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.512061 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.512539 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.512619 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260927 00:56:41.512908 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.508369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51265 (local address 127.0.98.2:36843)
0927 00:56:41.508485 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.508490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.508786 (+ 296us) server_negotiation.cc:407] Connection header received
0927 00:56:41.508987 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.508990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.509039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.509082 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:41.509515 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.510142 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.511611 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.512124 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.512219 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.512302 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.512706 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.512787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:56:41.513316 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.507951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36843 (local address 127.0.98.1:51265)
0927 00:56:41.508715 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.508728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.508873 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.509221 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.509224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.509240 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.509413 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.509421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.510289 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.510303 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.511466 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.511478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.512420 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.512423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.512557 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.512584 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.513038 (+ 454us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.513139 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":26,"spinlock_wait_cycles":29312,"thread_start_us":146,"threads_started":1}
I20260927 00:56:41.513620 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.507819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41275 (local address 127.0.98.1:54869)
0927 00:56:41.508611 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.508625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:41.508873 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:41.509282 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:41.509285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:41.509296 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:41.509543 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.509549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.510239 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.510242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.511194 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:41.511206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.512131 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.512217 (+ 86us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:41.512367 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.512395 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:56:41.513121 (+ 726us) client_negotiation.cc:240] Negotiation successful
0927 00:56:41.513234 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":92,"spinlock_wait_cycles":6272,"thread_start_us":103,"threads_started":1}
I20260927 00:56:41.513863 18370 raft_consensus.cc:2454] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12120ce7e49744ae99d165b6d2c9ed0b in term 1.
I20260927 00:56:41.513738 18108 raft_consensus.cc:2454] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12120ce7e49744ae99d165b6d2c9ed0b in term 1.
I20260927 00:56:41.514254 18173 leader_election.cc:302] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12120ce7e49744ae99d165b6d2c9ed0b, 736d2c575fce4b3ea4514132b22a6c14; no voters:
I20260927 00:56:41.514333 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75658ce282204e778f26b25d7f46a574" candidate_uuid: "59f54c61e1b64f93bf142a57d2995d74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" is_pre_election: true
I20260927 00:56:41.514478 18440 raft_consensus.cc:2781] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:41.514734 18440 raft_consensus.cc:686] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 1 LEADER]: Becoming Leader. State: Replica: 12120ce7e49744ae99d165b6d2c9ed0b, State: Running, Role: LEADER
I20260927 00:56:41.514930 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75658ce282204e778f26b25d7f46a574" candidate_uuid: "59f54c61e1b64f93bf142a57d2995d74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736d2c575fce4b3ea4514132b22a6c14" is_pre_election: true
I20260927 00:56:41.515185 18370 raft_consensus.cc:2379] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f54c61e1b64f93bf142a57d2995d74 in current term 1: Already voted for candidate 12120ce7e49744ae99d165b6d2c9ed0b in this term.
I20260927 00:56:41.515187 18440 consensus_queue.cc:237] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } }
I20260927 00:56:41.518035 18041 leader_election.cc:302] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f54c61e1b64f93bf142a57d2995d74; no voters: 12120ce7e49744ae99d165b6d2c9ed0b, 736d2c575fce4b3ea4514132b22a6c14
I20260927 00:56:41.518356 18439 raft_consensus.cc:2726] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:56:41.519225 17966 catalog_manager.cc:5495] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 12120ce7e49744ae99d165b6d2c9ed0b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12120ce7e49744ae99d165b6d2c9ed0b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36843 } health_report { overall_health: HEALTHY } } }
I20260927 00:56:41.560828 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.556958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.0.98.2:36843)
0927 00:56:41.557078 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:41.557083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:41.557099 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:56:41.557264 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:41.557266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:41.557321 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:41.557419 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:41.557646 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.558164 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.559100 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:41.559401 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:41.559503 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:41.559883 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:41.560420 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:41.560529 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:41.560608 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 00:56:41.560673 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 00:56:41.566489 18417 tablet.cc:2369] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:56:41.567740 18370 raft_consensus.cc:1261] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 1 FOLLOWER]: Refusing update from remote peer 12120ce7e49744ae99d165b6d2c9ed0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:41.568032 18108 raft_consensus.cc:1261] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74 [term 1 FOLLOWER]: Refusing update from remote peer 12120ce7e49744ae99d165b6d2c9ed0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:41.568385 18445 consensus_queue.cc:1035] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "736d2c575fce4b3ea4514132b22a6c14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:41.568651 18440 consensus_queue.cc:1035] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f54c61e1b64f93bf142a57d2995d74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:41.577324 18450 mvcc.cc:205] Tried to move back new op lower bound from 7333767584012668928 to 7333767583809482752. Current Snapshot: MvccSnapshot[applied={T|T < 7333767584012668928}]
I20260927 00:56:41.587739 17964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:41.588003 17964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:41.590970 17964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 271b588613f44d80af43e71abea7419a: Sending DeleteTablet for 3 replicas of tablet 75658ce282204e778f26b25d7f46a574
I20260927 00:56:41.591867 18350 tablet_service.cc:1514] Processing DeleteTablet for tablet 75658ce282204e778f26b25d7f46a574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:41 UTC) from {username='slave'} at 127.0.0.1:44366
I20260927 00:56:41.592283 18088 tablet_service.cc:1514] Processing DeleteTablet for tablet 75658ce282204e778f26b25d7f46a574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:41 UTC) from {username='slave'} at 127.0.0.1:42870
I20260927 00:56:41.592540 18219 tablet_service.cc:1514] Processing DeleteTablet for tablet 75658ce282204e778f26b25d7f46a574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:41 UTC) from {username='slave'} at 127.0.0.1:37194
I20260927 00:56:41.593010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18026
I20260927 00:56:41.593001 18459 tablet_replica.cc:330] T 75658ce282204e778f26b25d7f46a574 P 59f54c61e1b64f93bf142a57d2995d74: stopping tablet replica
I20260927 00:56:41.593103 18460 tablet_replica.cc:330] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: stopping tablet replica
I20260927 00:56:41.594610 18460 raft_consensus.cc:2229] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:56:41.595264 18460 raft_consensus.cc:2258] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:41.597265 18460 ts_tablet_manager.cc:1905] T 75658ce282204e778f26b25d7f46a574 P 12120ce7e49744ae99d165b6d2c9ed0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:41.599418 18458 tablet_replica.cc:330] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: stopping tablet replica
I20260927 00:56:41.599870 18458 raft_consensus.cc:2229] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:41.600421 18458 raft_consensus.cc:2258] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:41.600828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18157
W20260927 00:56:41.600903 17950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32797 (error 108)
I20260927 00:56:41.603304 18458 ts_tablet_manager.cc:1905] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:41.603634 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.603275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32797 (local address 127.0.0.1:42878)
0927 00:56:41.603463 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.603531 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 00:56:41.603808 17950 catalog_manager.cc:4584] TS 59f54c61e1b64f93bf142a57d2995d74 (127.0.98.1:32797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75658ce282204e778f26b25d7f46a574: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32797: connect: Connection refused (error 111)
I20260927 00:56:41.605871 18458 ts_tablet_manager.cc:1918] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:41.605962 18458 log.cc:1196] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/75658ce282204e778f26b25d7f46a574
I20260927 00:56:41.606221 18458 ts_tablet_manager.cc:1939] T 75658ce282204e778f26b25d7f46a574 P 736d2c575fce4b3ea4514132b22a6c14: Deleting consensus metadata
I20260927 00:56:41.607841 17951 catalog_manager.cc:4841] TS 736d2c575fce4b3ea4514132b22a6c14 (127.0.98.3:41275): tablet 75658ce282204e778f26b25d7f46a574 (table test-table [id=42f9b81d96d04e74bd0ec138d740c6cb]) successfully deleted
I20260927 00:56:41.610502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18288
I20260927 00:56:41.610693 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:41.610317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36843 (local address 127.0.0.1:37212)
0927 00:56:41.610524 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:41.610586 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 00:56:41.617973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 17935
2026-09-27T00:56:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1228 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:41 dist-test-slave-9fh2 krb5kdc[18464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:41 dist-test-slave-9fh2 krb5kdc[18464](info): set up 2 sockets
Sep 27 00:56:41 dist-test-slave-9fh2 krb5kdc[18464](info): commencing operation
krb5kdc: starting...
W20260927 00:56:42.595129 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:42 dist-test-slave-9fh2 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:56:42.728776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36143
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:42.844938 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:42.845237 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:42.845278 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:42.849313 18477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:42.849380 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:42.849399 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:42.849414 18477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:42.849428 18477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:42.854800 18477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:42.855924 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:42.857056 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:42.866195 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:42.866284 18477 server_base.cc:1034] running on GCE node
W20260927 00:56:42.866518 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:42.866210 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:42.867472 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:42.868948 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:42.870149 18477 hybrid_clock.cc:648] HybridClock initialized: now 1790470602870118 us; error 51 us; skew 500 ppm
Sep 27 00:56:42 dist-test-slave-9fh2 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:42.873579 18477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:56:42.874782 18477 webserver.cc:460] Webserver started at http://127.0.98.62:41997/ using document root <none> and password file <none>
I20260927 00:56:42.875505 18477 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:42.875581 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:42.875833 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:42.878463 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "2a6b018bc1994cfda93df61a4db2ec77"
format_stamp: "Formatted at 2026-09-27 00:56:42 on dist-test-slave-9fh2"
server_key: "3daf1feb3109fd8e5ec0993230c3f643"
server_key_iv: "54fde54390aa937005efe5debd6e51b9"
server_key_version: "encryptionkey@0"
I20260927 00:56:42.879117 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a6b018bc1994cfda93df61a4db2ec77"
format_stamp: "Formatted at 2026-09-27 00:56:42 on dist-test-slave-9fh2"
server_key: "3daf1feb3109fd8e5ec0993230c3f643"
server_key_iv: "54fde54390aa937005efe5debd6e51b9"
server_key_version: "encryptionkey@0"
I20260927 00:56:42.883790 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:56:42.886951 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:42.888830 18477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:42.889035 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "2a6b018bc1994cfda93df61a4db2ec77"
format_stamp: "Formatted at 2026-09-27 00:56:42 on dist-test-slave-9fh2"
server_key: "3daf1feb3109fd8e5ec0993230c3f643"
server_key_iv: "54fde54390aa937005efe5debd6e51b9"
server_key_version: "encryptionkey@0"
I20260927 00:56:42.889199 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:42.903234 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:42.907019 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:42.907269 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:42.915738 18477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36143
I20260927 00:56:42.915771 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36143 every 8 connection(s)
I20260927 00:56:42.916919 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:42.919974 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:42.925554 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: Bootstrap starting.
I20260927 00:56:42.927407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18477
I20260927 00:56:42.927557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:42.927879 392 external_mini_cluster.cc:1428] Setting key 178535c11b23d7a474eab3181ae9dc69
I20260927 00:56:42.928093 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:42.928982 18545 log.cc:824] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:42.930951 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: No bootstrap required, opened a new log
I20260927 00:56:42.935274 18545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b018bc1994cfda93df61a4db2ec77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36143 } }
I20260927 00:56:42.935542 18545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:42.935609 18545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6b018bc1994cfda93df61a4db2ec77, State: Initialized, Role: FOLLOWER
I20260927 00:56:42.936045 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b018bc1994cfda93df61a4db2ec77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36143 } }
I20260927 00:56:42.936195 18545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:42.936268 18545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:42.936357 18545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:42.937580 18545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b018bc1994cfda93df61a4db2ec77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36143 } }
I20260927 00:56:42.937988 18545 leader_election.cc:302] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a6b018bc1994cfda93df61a4db2ec77; no voters:
I20260927 00:56:42.938314 18545 leader_election.cc:288] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:42.938512 18550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 00:56:42 dist-test-slave-9fh2 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:42.939059 18550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [term 1 LEADER]: Becoming Leader. State: Replica: 2a6b018bc1994cfda93df61a4db2ec77, State: Running, Role: LEADER
I20260927 00:56:42.939564 18550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b018bc1994cfda93df61a4db2ec77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36143 } }
I20260927 00:56:42.939774 18545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:42.941689 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a6b018bc1994cfda93df61a4db2ec77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b018bc1994cfda93df61a4db2ec77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36143 } } }
I20260927 00:56:42.941851 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:42.942210 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6b018bc1994cfda93df61a4db2ec77. Latest consensus state: current_term: 1 leader_uuid: "2a6b018bc1994cfda93df61a4db2ec77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b018bc1994cfda93df61a4db2ec77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36143 } } }
I20260927 00:56:42.942318 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77 [sys.catalog]: This master's current role is: LEADER
W20260927 00:56:42.945099 18564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:42.945179 18564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:42.945257 18565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:42.945959 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:42.929464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.98.62:36143)
0927 00:56:42.929968 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:42.929987 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:42.930030 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:56:42.930913 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:42.930936 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:42.931433 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:42.931807 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:42.932787 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:42.934029 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:42.935299 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:42.936018 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:42.936189 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:42.939445 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:42.939452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:42.939471 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:42.939516 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:42.941827 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:42.942518 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:42.942524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:42.942530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:42.942678 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:42.943100 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:42.943103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:42.943105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:42.943666 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:42.943865 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:42.944603 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 00:56:42.944913 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":134,"threads_started":1}
I20260927 00:56:42.946100 18565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:42.952157 18565 catalog_manager.cc:1347] Generated new cluster ID: d4bbf56e55f24129bdac4b5113d8b470
I20260927 00:56:42.952234 18565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:42.970463 18565 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:42.971836 18565 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:42.984737 18565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: Generated new TSK 0
I20260927 00:56:42.985684 18565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:56:43.020112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36143
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:43.150765 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:43.151165 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:43.151237 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:43.155541 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:43.155634 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:43.155740 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:43.161943 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:43.163561 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:43.164928 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:43.177042 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:43.177052 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:43.177163 18571 server_base.cc:1034] running on GCE node
W20260927 00:56:43.177052 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:43.177978 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:43.178843 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:43.180047 18571 hybrid_clock.cc:648] HybridClock initialized: now 1790470603180020 us; error 47 us; skew 500 ppm
Sep 27 00:56:43 dist-test-slave-9fh2 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:43.183432 18571 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:56:43.184672 18571 webserver.cc:460] Webserver started at http://127.0.98.1:41413/ using document root <none> and password file <none>
I20260927 00:56:43.185376 18571 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:43.185451 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:43.185679 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:43.188385 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a67f23a5c634c3585a43add3305456b"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "1b012cbf29d6db822437bc5f9d918615"
server_key_iv: "870a034a0a2c6a691a68690eb62f1044"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.189044 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a67f23a5c634c3585a43add3305456b"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "1b012cbf29d6db822437bc5f9d918615"
server_key_iv: "870a034a0a2c6a691a68690eb62f1044"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.193794 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:43.196995 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.198706 18571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:56:43.198860 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "8a67f23a5c634c3585a43add3305456b"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "1b012cbf29d6db822437bc5f9d918615"
server_key_iv: "870a034a0a2c6a691a68690eb62f1044"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.198998 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:43.214120 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:43.217906 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:43.218128 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:43.218847 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:43.219937 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:43.219990 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.220052 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:43.220077 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.230888 18571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40137
I20260927 00:56:43.230933 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40137 every 8 connection(s)
I20260927 00:56:43.232054 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:43.238670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18571
I20260927 00:56:43.238803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:56:43.239133 392 external_mini_cluster.cc:1428] Setting key 312b069503fcf1a80e1d9675b7bbac3f
Sep 27 00:56:43 dist-test-slave-9fh2 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:43.248381 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.234392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42871 (local address 127.0.98.62:36143)
0927 00:56:43.234582 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.234587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.235618 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 00:56:43.236633 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.236636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.236706 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.236779 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:43.238170 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.238745 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.240138 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.240375 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.240490 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.243824 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:43.243828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:43.243831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:43.243865 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:43.245709 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.246381 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.246384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.246386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.246478 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.246935 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.246939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.246941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.247221 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:43.247323 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.247999 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.248179 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:56:43.249493 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.234833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36143 (local address 127.0.98.1:42871)
0927 00:56:43.235434 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:43.235486 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:43.236407 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:43.236975 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:43.236982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:43.237387 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:43.238018 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.238037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.238868 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.238871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.240010 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.240026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.240607 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.240613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.240976 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.241784 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:43.241811 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:43.243585 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:43.245868 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:43.245875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:43.245893 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:43.246246 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:43.246567 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:43.246569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:43.246571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:43.246753 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:43.247357 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:43.247365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:43.247714 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 00:56:43.247922 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 00:56:43.248381 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":133,"threads_started":1}
I20260927 00:56:43.251016 18700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:56:43.251461 18700 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:43.252390 18700 heartbeater.cc:511] Master 127.0.98.62:36143 requested a full tablet report, sending...
I20260927 00:56:43.254585 18509 ts_manager.cc:194] Registered new tserver with Master: 8a67f23a5c634c3585a43add3305456b (127.0.98.1:40137)
I20260927 00:56:43.256045 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:56:43.268688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36143
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:43.386173 18705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:43.386502 18705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:43.386547 18705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:43.390695 18705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:43.390775 18705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:43.390837 18705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:43.396426 18705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:43.397576 18705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:43.398633 18705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:43.409193 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:43.409363 18705 server_base.cc:1034] running on GCE node
W20260927 00:56:43.409204 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:43.409216 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:43.410168 18705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:43.411036 18705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:43.412209 18705 hybrid_clock.cc:648] HybridClock initialized: now 1790470603412172 us; error 52 us; skew 500 ppm
Sep 27 00:56:43 dist-test-slave-9fh2 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:43.415426 18705 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:56:43.416496 18705 webserver.cc:460] Webserver started at http://127.0.98.2:39173/ using document root <none> and password file <none>
I20260927 00:56:43.417213 18705 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:43.417285 18705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:43.417518 18705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:43.420311 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "8217513924e345b1a6771eebe9668762"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "a133d18c74dc3ed69cfb66861e0afc50"
server_key_iv: "635069866ac08374c546084a33f79918"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.421053 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8217513924e345b1a6771eebe9668762"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "a133d18c74dc3ed69cfb66861e0afc50"
server_key_iv: "635069866ac08374c546084a33f79918"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.425793 18705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:43.429075 18720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.430943 18705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:56:43.431099 18705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "8217513924e345b1a6771eebe9668762"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "a133d18c74dc3ed69cfb66861e0afc50"
server_key_iv: "635069866ac08374c546084a33f79918"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.431322 18705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:43.452795 18705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:43.456440 18705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:43.456655 18705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:43.457345 18705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:43.458346 18705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:43.458405 18705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.458454 18705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:43.458487 18705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.469435 18705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43949
I20260927 00:56:43.469480 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43949 every 8 connection(s)
I20260927 00:56:43.470518 18705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:43.477000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18705
I20260927 00:56:43.477147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:56:43.477552 392 external_mini_cluster.cc:1428] Setting key 8b19fba65ef614fcb6d14cac3420d67a
Sep 27 00:56:43 dist-test-slave-9fh2 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:43.486253 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.472583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41145 (local address 127.0.98.62:36143)
0927 00:56:43.472747 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.472751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.473631 (+ 880us) server_negotiation.cc:407] Connection header received
0927 00:56:43.474717 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.474719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.474777 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.474865 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:43.476325 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.476879 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.478464 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.478742 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.478830 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.481981 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:43.481986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:43.481988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:43.482020 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:43.483543 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.484409 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.484413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.484415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.484571 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.485006 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.485010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.485013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.485236 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:43.485331 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.485911 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.486059 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
I20260927 00:56:43.487344 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.472892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36143 (local address 127.0.98.2:41145)
0927 00:56:43.473428 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:43.473484 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:43.474477 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:43.475105 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:43.475113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:43.475559 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:43.476147 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.476163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.477045 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.477048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.478339 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.478350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.478921 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.478925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.479110 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.479877 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:43.479901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:43.481767 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:43.483848 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:43.483855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:43.483883 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:43.484264 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:43.484681 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:43.484684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:43.484686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:43.484899 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:43.485344 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:43.485352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:43.485655 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 00:56:43.485851 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 00:56:43.486248 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":139,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:56:43.488924 18834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36143
I20260927 00:56:43.489210 18834 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:43.489838 18834 heartbeater.cc:511] Master 127.0.98.62:36143 requested a full tablet report, sending...
I20260927 00:56:43.491380 18509 ts_manager.cc:194] Registered new tserver with Master: 8217513924e345b1a6771eebe9668762 (127.0.98.2:43949)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:56:43.492120 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:56:43.510694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36143
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:43.621706 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:43.621965 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:43.622007 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:43.625991 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:43.626067 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:43.626127 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:43.631364 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:43.632414 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:43.633402 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:43.644002 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:43.644131 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:43.644635 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:43.644793 18839 server_base.cc:1034] running on GCE node
I20260927 00:56:43.645267 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:43.646066 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:43.647329 18839 hybrid_clock.cc:648] HybridClock initialized: now 1790470603647294 us; error 63 us; skew 500 ppm
Sep 27 00:56:43 dist-test-slave-9fh2 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:43.650216 18839 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:56:43.651173 18839 webserver.cc:460] Webserver started at http://127.0.98.3:38521/ using document root <none> and password file <none>
I20260927 00:56:43.651762 18839 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:43.651826 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:43.651994 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:43.654013 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "cba7abbf73b5401aaee400e160937ed3"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "720df25c780cb90115fe44fc4a216359"
server_key_iv: "a7712ddc4313f06d6b495ddf7cb95ace"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.654522 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cba7abbf73b5401aaee400e160937ed3"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "720df25c780cb90115fe44fc4a216359"
server_key_iv: "a7712ddc4313f06d6b495ddf7cb95ace"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.658680 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:56:43.661453 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.662941 18839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:56:43.663057 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "cba7abbf73b5401aaee400e160937ed3"
format_stamp: "Formatted at 2026-09-27 00:56:43 on dist-test-slave-9fh2"
server_key: "720df25c780cb90115fe44fc4a216359"
server_key_iv: "a7712ddc4313f06d6b495ddf7cb95ace"
server_key_version: "encryptionkey@0"
I20260927 00:56:43.663229 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:43.681888 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:43.684999 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:43.685166 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:43.685727 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:43.686559 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:43.686611 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.686652 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:43.686667 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:43.696825 18839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43265
I20260927 00:56:43.696986 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43265 every 8 connection(s)
I20260927 00:56:43.697862 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:43.698330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 18839
I20260927 00:56:43.698467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:56:43.698833 392 external_mini_cluster.cc:1428] Setting key 5827d8765226932b3fd46ed6600b4973
Sep 27 00:56:43 dist-test-slave-9fh2 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:43.713601 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.699816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57009 (local address 127.0.98.62:36143)
0927 00:56:43.700018 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.700025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.701568 (+ 1543us) server_negotiation.cc:407] Connection header received
0927 00:56:43.702809 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.702812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.702878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.702955 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:43.704481 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.705079 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.706295 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.706542 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.706633 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.709660 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:43.709664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:43.709666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:43.709697 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:43.711074 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.711681 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.711684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.711685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.711767 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.712244 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.712246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.712248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.712485 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:43.712583 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.713264 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.713425 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:56:43.714407 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.700272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36143 (local address 127.0.98.3:57009)
0927 00:56:43.701358 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:43.701453 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:43.702573 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:43.703206 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:43.703216 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:43.703705 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:43.704331 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.704348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.705207 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.705210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.706178 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.706189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.706708 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.706712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.706894 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.707544 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:43.707562 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:43.709428 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:43.711243 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:43.711248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:43.711262 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:43.711584 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:43.711934 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:43.711937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:43.711939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:43.712130 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:43.712599 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:43.712605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:43.712961 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 00:56:43.713140 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:56:43.713516 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":149,"threads_started":1}
I20260927 00:56:43.715581 18968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36143
I20260927 00:56:43.715927 18968 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:43.716568 18968 heartbeater.cc:511] Master 127.0.98.62:36143 requested a full tablet report, sending...
I20260927 00:56:43.717746 18509 ts_manager.cc:194] Registered new tserver with Master: cba7abbf73b5401aaee400e160937ed3 (127.0.98.3:43265)
I20260927 00:56:43.718422 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57009
I20260927 00:56:43.726100 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:43.736330 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.728512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.62:36143)
0927 00:56:43.728738 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.728743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.728855 (+ 112us) server_negotiation.cc:407] Connection header received
0927 00:56:43.729014 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.729017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.729064 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.729140 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:43.729865 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.730400 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.731266 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.731473 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.731559 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.733113 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:43.733116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:43.733117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:43.733139 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:43.734454 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.734926 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.734929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.734930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.734997 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:43.735336 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:43.735339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:43.735341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:43.735542 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:43.735688 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.736125 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.736207 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:56:43.739691 18509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:43.743026 18509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:43.759639 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.753210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.0.98.1:40137)
0927 00:56:43.754420 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.754429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.754458 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:56:43.754570 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.754574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.754803 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.754965 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:43.755452 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.756688 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.758226 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.758955 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.759066 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.759238 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.759363 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.759465 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":102,"threads_started":1}
I20260927 00:56:43.759685 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.753054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40137 (local address 127.0.0.1:50346)
0927 00:56:43.753681 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:43.753733 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:43.753856 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:43.755103 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:43.755106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:43.755120 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:43.755329 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.755336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.756823 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.756826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.758090 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.758101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.759089 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.759092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.759200 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.759220 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:56:43.759324 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:56:43.759489 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":86,"thread_start_us":100,"threads_started":1}
I20260927 00:56:43.760336 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.753343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43265 (local address 127.0.0.1:57242)
0927 00:56:43.754023 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:43.754041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:43.754148 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:43.755137 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:43.755141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:43.755197 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:43.755378 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.755385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.756862 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.756866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.757823 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.757832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.758909 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.758912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.759006 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.759059 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 00:56:43.760102 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0927 00:56:43.760200 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":28,"spinlock_wait_cycles":2304,"thread_start_us":120,"threads_started":1}
I20260927 00:56:43.760492 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.753940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.0.98.3:43265)
0927 00:56:43.754454 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.754464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.754482 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:56:43.754565 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.754570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.754803 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.754965 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:43.755527 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.756714 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.757941 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.758707 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.758826 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.758983 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.760238 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.760339 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":122,"threads_started":1}
I20260927 00:56:43.760499 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.753860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43949 (local address 127.0.0.1:50760)
0927 00:56:43.754185 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:43.754198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:43.754282 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:43.754942 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:43.754948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:43.754970 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:43.755100 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.755106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.756508 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.756511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.758090 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:43.758101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.760036 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.760040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:43.760152 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.760194 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:56:43.760267 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:56:43.760363 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":33,"thread_start_us":106,"threads_started":1}
I20260927 00:56:43.760931 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:43.754243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.0.98.2:43949)
0927 00:56:43.754542 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:43.754556 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:43.754574 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:56:43.754635 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:43.754639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:43.754804 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:43.754933 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:43.755257 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.756337 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.759067 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:43.759904 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:43.760019 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:43.760148 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:43.760723 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 00:56:43.760814 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
I20260927 00:56:43.763324 18634 tablet_service.cc:1467] Processing CreateTablet for tablet e671eaecd073454fad07dc03f556bcfc (DEFAULT_TABLE table=test-table [id=2cfb2584786c4bfa9c36b097d29c5ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:43.764074 18768 tablet_service.cc:1467] Processing CreateTablet for tablet e671eaecd073454fad07dc03f556bcfc (DEFAULT_TABLE table=test-table [id=2cfb2584786c4bfa9c36b097d29c5ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:43.764523 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e671eaecd073454fad07dc03f556bcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:43.765077 18768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e671eaecd073454fad07dc03f556bcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:43.765221 18902 tablet_service.cc:1467] Processing CreateTablet for tablet e671eaecd073454fad07dc03f556bcfc (DEFAULT_TABLE table=test-table [id=2cfb2584786c4bfa9c36b097d29c5ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:43.766180 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e671eaecd073454fad07dc03f556bcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:43.774307 18985 tablet_bootstrap.cc:492] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: Bootstrap starting.
I20260927 00:56:43.776687 18985 tablet_bootstrap.cc:654] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:43.777565 18985 log.cc:824] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:43.778975 18986 tablet_bootstrap.cc:492] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Bootstrap starting.
I20260927 00:56:43.779634 18985 tablet_bootstrap.cc:492] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: No bootstrap required, opened a new log
I20260927 00:56:43.779829 18985 ts_tablet_manager.cc:1397] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:43.782568 18986 tablet_bootstrap.cc:654] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:43.783119 18988 tablet_bootstrap.cc:492] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Bootstrap starting.
I20260927 00:56:43.783289 18985 raft_consensus.cc:348] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:43.783547 18985 raft_consensus.cc:374] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:43.783598 18985 raft_consensus.cc:729] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a67f23a5c634c3585a43add3305456b, State: Initialized, Role: FOLLOWER
I20260927 00:56:43.783587 18986 log.cc:824] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:43.784068 18985 consensus_queue.cc:260] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:43.785037 18985 ts_tablet_manager.cc:1428] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:56:43.785389 18700 heartbeater.cc:503] Master 127.0.98.62:36143 was elected leader, sending a full tablet report...
I20260927 00:56:43.785851 18988 tablet_bootstrap.cc:654] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:43.786062 18986 tablet_bootstrap.cc:492] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: No bootstrap required, opened a new log
I20260927 00:56:43.786305 18986 ts_tablet_manager.cc:1397] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:56:43.788065 18988 log.cc:824] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:43.790210 18988 tablet_bootstrap.cc:492] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: No bootstrap required, opened a new log
I20260927 00:56:43.790403 18988 ts_tablet_manager.cc:1397] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 00:56:43.790954 18986 raft_consensus.cc:348] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:43.791345 18986 raft_consensus.cc:374] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:43.791397 18986 raft_consensus.cc:729] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8217513924e345b1a6771eebe9668762, State: Initialized, Role: FOLLOWER
I20260927 00:56:43.792155 18986 consensus_queue.cc:260] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:43.793532 18986 ts_tablet_manager.cc:1428] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:56:43.793848 18834 heartbeater.cc:503] Master 127.0.98.62:36143 was elected leader, sending a full tablet report...
I20260927 00:56:43.793814 18988 raft_consensus.cc:348] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:43.794100 18988 raft_consensus.cc:374] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:43.794147 18988 raft_consensus.cc:729] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba7abbf73b5401aaee400e160937ed3, State: Initialized, Role: FOLLOWER
I20260927 00:56:43.794613 18988 consensus_queue.cc:260] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:43.795703 18968 heartbeater.cc:503] Master 127.0.98.62:36143 was elected leader, sending a full tablet report...
I20260927 00:56:43.795753 18988 ts_tablet_manager.cc:1428] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260927 00:56:43.949186 18969 tablet.cc:2369] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:56:43.971937 18835 tablet.cc:2369] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:56:43.983693 18701 tablet.cc:2369] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:56:44.177150 18992 raft_consensus.cc:482] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:44.177440 18992 raft_consensus.cc:504] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:44.178957 18992 leader_election.cc:288] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a67f23a5c634c3585a43add3305456b (127.0.98.1:40137), cba7abbf73b5401aaee400e160937ed3 (127.0.98.3:43265)
I20260927 00:56:44.183346 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.179317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43265 (local address 127.0.98.2:45175)
0927 00:56:44.179641 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:44.179658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:44.179805 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:44.180094 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:44.180097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:44.180117 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:44.180312 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:44.180320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.181088 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.181091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:44.182030 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:44.182040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.182788 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.182792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:44.182901 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.182937 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:56:44.183029 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:56:44.183120 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260927 00:56:44.183377 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.179430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45175 (local address 127.0.98.3:43265)
0927 00:56:44.179726 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:44.179731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:44.179756 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:56:44.179935 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:44.179938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:44.179991 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:44.180059 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:44.180431 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.180961 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.182168 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.182663 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.182744 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.182803 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:44.183094 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 00:56:44.183228 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260927 00:56:44.183808 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.179430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33655 (local address 127.0.98.1:40137)
0927 00:56:44.179613 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:44.179618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:44.179863 (+ 245us) server_negotiation.cc:407] Connection header received
0927 00:56:44.180021 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:44.180023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:44.180080 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:44.180161 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:44.180538 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.181196 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.182463 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.183059 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.183194 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.183276 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:44.183549 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 00:56:44.183712 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:56:44.184414 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.179317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40137 (local address 127.0.98.2:33655)
0927 00:56:44.179749 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:44.179765 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:44.179899 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:44.180204 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:44.180207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:44.180225 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:44.180373 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:44.180380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.181328 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.181331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:44.182329 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:44.182340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.183220 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.183224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:44.183337 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.183370 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:56:44.184144 (+ 774us) client_negotiation.cc:240] Negotiation successful
0927 00:56:44.184255 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":29,"thread_start_us":86,"threads_started":1}
I20260927 00:56:44.184326 18922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e671eaecd073454fad07dc03f556bcfc" candidate_uuid: "8217513924e345b1a6771eebe9668762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba7abbf73b5401aaee400e160937ed3" is_pre_election: true
I20260927 00:56:44.184696 18922 raft_consensus.cc:2454] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8217513924e345b1a6771eebe9668762 in term 0.
I20260927 00:56:44.185248 18722 leader_election.cc:302] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8217513924e345b1a6771eebe9668762, cba7abbf73b5401aaee400e160937ed3; no voters:
I20260927 00:56:44.185293 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e671eaecd073454fad07dc03f556bcfc" candidate_uuid: "8217513924e345b1a6771eebe9668762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a67f23a5c634c3585a43add3305456b" is_pre_election: true
I20260927 00:56:44.185549 18992 raft_consensus.cc:2781] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:44.185678 18992 raft_consensus.cc:482] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:44.185662 18654 raft_consensus.cc:2454] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8217513924e345b1a6771eebe9668762 in term 0.
I20260927 00:56:44.185724 18992 raft_consensus.cc:3037] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:44.187114 18992 raft_consensus.cc:504] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:44.187737 18992 leader_election.cc:288] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [CANDIDATE]: Term 1 election: Requested vote from peers 8a67f23a5c634c3585a43add3305456b (127.0.98.1:40137), cba7abbf73b5401aaee400e160937ed3 (127.0.98.3:43265)
I20260927 00:56:44.188086 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e671eaecd073454fad07dc03f556bcfc" candidate_uuid: "8217513924e345b1a6771eebe9668762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a67f23a5c634c3585a43add3305456b"
I20260927 00:56:44.188234 18654 raft_consensus.cc:3037] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:44.188292 18922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e671eaecd073454fad07dc03f556bcfc" candidate_uuid: "8217513924e345b1a6771eebe9668762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba7abbf73b5401aaee400e160937ed3"
I20260927 00:56:44.188436 18922 raft_consensus.cc:3037] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:44.189540 18654 raft_consensus.cc:2454] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8217513924e345b1a6771eebe9668762 in term 1.
I20260927 00:56:44.189666 18922 raft_consensus.cc:2454] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8217513924e345b1a6771eebe9668762 in term 1.
I20260927 00:56:44.189963 18721 leader_election.cc:302] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8217513924e345b1a6771eebe9668762, 8a67f23a5c634c3585a43add3305456b; no voters:
I20260927 00:56:44.190160 18992 raft_consensus.cc:2781] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:44.190418 18992 raft_consensus.cc:686] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 1 LEADER]: Becoming Leader. State: Replica: 8217513924e345b1a6771eebe9668762, State: Running, Role: LEADER
I20260927 00:56:44.190825 18992 consensus_queue.cc:237] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } }
I20260927 00:56:44.194813 18509 catalog_manager.cc:5495] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8217513924e345b1a6771eebe9668762 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8217513924e345b1a6771eebe9668762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8217513924e345b1a6771eebe9668762" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 } health_report { overall_health: UNKNOWN } } }
I20260927 00:56:44.216357 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.212330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.0.98.2:43949)
0927 00:56:44.212480 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:44.212484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:44.212501 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:56:44.212593 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:44.212595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:44.212656 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:44.212802 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:44.213040 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.213615 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.214610 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.214844 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.214943 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.215371 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:44.215925 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:44.216053 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:44.216137 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 00:56:44.216181 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:56:44.223901 18654 raft_consensus.cc:1261] T e671eaecd073454fad07dc03f556bcfc P 8a67f23a5c634c3585a43add3305456b [term 1 FOLLOWER]: Refusing update from remote peer 8217513924e345b1a6771eebe9668762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:44.223901 18922 raft_consensus.cc:1261] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 1 FOLLOWER]: Refusing update from remote peer 8217513924e345b1a6771eebe9668762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:44.224884 18996 consensus_queue.cc:1035] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cba7abbf73b5401aaee400e160937ed3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:44.225276 18992 consensus_queue.cc:1035] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a67f23a5c634c3585a43add3305456b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:44.234527 19000 mvcc.cc:205] Tried to move back new op lower bound from 7333767594892120064 to 7333767594768834560. Current Snapshot: MvccSnapshot[applied={T|T < 7333767594892120064}]
I20260927 00:56:44.243301 18999 mvcc.cc:205] Tried to move back new op lower bound from 7333767594892120064 to 7333767594768834560. Current Snapshot: MvccSnapshot[applied={T|T < 7333767594892120064 or (T in {7333767594892120064})}]
I20260927 00:56:44.245756 18509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:44.245990 18509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:44.249176 18509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a6b018bc1994cfda93df61a4db2ec77: Sending DeleteTablet for 3 replicas of tablet e671eaecd073454fad07dc03f556bcfc
I20260927 00:56:44.250195 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet e671eaecd073454fad07dc03f556bcfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50346
I20260927 00:56:44.250267 18902 tablet_service.cc:1514] Processing DeleteTablet for tablet e671eaecd073454fad07dc03f556bcfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57242
I20260927 00:56:44.250360 18768 tablet_service.cc:1514] Processing DeleteTablet for tablet e671eaecd073454fad07dc03f556bcfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50760
I20260927 00:56:44.250857 19010 tablet_replica.cc:330] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: stopping tablet replica
I20260927 00:56:44.251334 19010 raft_consensus.cc:2229] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:44.251538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18571
I20260927 00:56:44.251776 19010 raft_consensus.cc:2258] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:44.254882 19010 ts_tablet_manager.cc:1905] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:44.255373 19011 tablet_replica.cc:330] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: stopping tablet replica
I20260927 00:56:44.255825 19011 raft_consensus.cc:2229] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:56:44.256387 19011 raft_consensus.cc:2258] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:44.257886 19010 ts_tablet_manager.cc:1918] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:44.258011 19010 log.cc:1196] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/e671eaecd073454fad07dc03f556bcfc
I20260927 00:56:44.258311 19010 ts_tablet_manager.cc:1939] T e671eaecd073454fad07dc03f556bcfc P cba7abbf73b5401aaee400e160937ed3: Deleting consensus metadata
I20260927 00:56:44.258662 19011 ts_tablet_manager.cc:1905] T e671eaecd073454fad07dc03f556bcfc P 8217513924e345b1a6771eebe9668762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:44.259792 18494 catalog_manager.cc:4841] TS cba7abbf73b5401aaee400e160937ed3 (127.0.98.3:43265): tablet e671eaecd073454fad07dc03f556bcfc (table test-table [id=2cfb2584786c4bfa9c36b097d29c5ab1]) successfully deleted
W20260927 00:56:44.260295 18493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40137 (error 108)
I20260927 00:56:44.260509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18705
I20260927 00:56:44.262773 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.262335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40137 (local address 127.0.0.1:50354)
0927 00:56:44.262594 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:44.262672 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
W20260927 00:56:44.262938 18493 catalog_manager.cc:4584] TS 8a67f23a5c634c3585a43add3305456b (127.0.98.1:40137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e671eaecd073454fad07dc03f556bcfc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40137: connect: Connection refused (error 111)
I20260927 00:56:44.267684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18839
I20260927 00:56:44.268888 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.267821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43949 (local address 127.0.0.1:50786)
0927 00:56:44.268737 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:44.268802 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":816}
I20260927 00:56:44.273979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 18477
2026-09-27T00:56:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2656 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T00:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:44Z Disabled control of system clock
I20260927 00:56:44.307271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35239
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:56:44.439476 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:44.439854 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:44.439894 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:44.444520 19017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:44.444617 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:44.444633 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:44.444649 19017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:44.444664 19017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:44.450078 19017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:44.451683 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:44.453521 19017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:44.463847 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:44.463943 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:44.463868 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:44.464313 19017 server_base.cc:1034] running on GCE node
I20260927 00:56:44.465026 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:44.466436 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:44.467636 19017 hybrid_clock.cc:648] HybridClock initialized: now 1790470604467618 us; error 39 us; skew 500 ppm
I20260927 00:56:44.469995 19017 webserver.cc:460] Webserver started at http://127.0.98.62:45497/ using document root <none> and password file <none>
I20260927 00:56:44.470687 19017 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:44.470754 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:44.470978 19017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:44.473558 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "778304831a31453392756edbb4381e33"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "5698946ca3de21c3e358712da5f8dc99"
server_key_iv: "6ba0e3a74f828a573b13f7de0128fe3a"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.474224 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "778304831a31453392756edbb4381e33"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "5698946ca3de21c3e358712da5f8dc99"
server_key_iv: "6ba0e3a74f828a573b13f7de0128fe3a"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.478869 19017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:56:44.482110 19031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:44.483764 19017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:56:44.483980 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "778304831a31453392756edbb4381e33"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "5698946ca3de21c3e358712da5f8dc99"
server_key_iv: "6ba0e3a74f828a573b13f7de0128fe3a"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.484136 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:44.496615 19017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:44.497519 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:44.497717 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:44.506163 19017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35239
I20260927 00:56:44.506201 19083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35239 every 8 connection(s)
I20260927 00:56:44.507432 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:44.511134 19084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:44.514992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19017
I20260927 00:56:44.515184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:44.515511 392 external_mini_cluster.cc:1428] Setting key 7cb2be4689f40be9c9725b078fd2f6b3
I20260927 00:56:44.517469 19084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: Bootstrap starting.
I20260927 00:56:44.519891 19084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:44.520774 19084 log.cc:824] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:44.522924 19084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: No bootstrap required, opened a new log
I20260927 00:56:44.526907 19084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778304831a31453392756edbb4381e33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35239 } }
I20260927 00:56:44.527176 19084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:44.527243 19084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778304831a31453392756edbb4381e33, State: Initialized, Role: FOLLOWER
I20260927 00:56:44.527418 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.517480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.0.98.62:35239)
0927 00:56:44.518376 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:44.518395 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:44.518444 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:56:44.519341 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:44.519364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:44.519897 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:44.520316 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:44.520787 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.521919 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.523203 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.523789 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.523949 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.525002 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:44.525009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:44.525025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:44.525542 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:44.525615 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:44.525639 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:44.525722 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:44.525981 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 00:56:44.526331 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":183,"threads_started":1}
I20260927 00:56:44.527719 19084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778304831a31453392756edbb4381e33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35239 } }
I20260927 00:56:44.527863 19084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:44.527930 19084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:44.528024 19084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:44.529049 19084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778304831a31453392756edbb4381e33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35239 } }
I20260927 00:56:44.529421 19084 leader_election.cc:302] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778304831a31453392756edbb4381e33; no voters:
I20260927 00:56:44.529771 19084 leader_election.cc:288] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:44.529932 19089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:44.530215 19089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [term 1 LEADER]: Becoming Leader. State: Replica: 778304831a31453392756edbb4381e33, State: Running, Role: LEADER
I20260927 00:56:44.530609 19089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778304831a31453392756edbb4381e33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35239 } }
I20260927 00:56:44.531003 19084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:44.532732 19090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778304831a31453392756edbb4381e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778304831a31453392756edbb4381e33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35239 } } }
I20260927 00:56:44.532894 19090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:44.532708 19091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778304831a31453392756edbb4381e33. Latest consensus state: current_term: 1 leader_uuid: "778304831a31453392756edbb4381e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778304831a31453392756edbb4381e33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35239 } } }
I20260927 00:56:44.533741 19091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33 [sys.catalog]: This master's current role is: LEADER
W20260927 00:56:44.536137 19104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:44.536232 19104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:44.536350 19103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:44.537401 19103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:44.544268 19103 catalog_manager.cc:1347] Generated new cluster ID: db2d149ea9f94f2b99becdce75773470
I20260927 00:56:44.544363 19103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:44.560369 19103 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:44.561602 19103 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:44.569321 19103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: Generated new TSK 0
I20260927 00:56:44.570297 19103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:44.578171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35239
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:56:44.710508 19108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:44.710865 19108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:44.710908 19108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:44.715436 19108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:44.715518 19108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:44.715592 19108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:44.721176 19108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:44.722704 19108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:44.724462 19108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:44.736270 19113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:44.736275 19116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:44.736361 19108 server_base.cc:1034] running on GCE node
W20260927 00:56:44.736642 19114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:44.737133 19108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:44.737980 19108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:44.739184 19108 hybrid_clock.cc:648] HybridClock initialized: now 1790470604739158 us; error 42 us; skew 500 ppm
I20260927 00:56:44.741533 19108 webserver.cc:460] Webserver started at http://127.0.98.1:46587/ using document root <none> and password file <none>
I20260927 00:56:44.742218 19108 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:44.742288 19108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:44.742491 19108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:44.744925 19108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa97c35a63db4d6093b2d80fb2be432c"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "ecd7b5ace43a211d8c9d0feec6c88d37"
server_key_iv: "abd253dfaddb8645ccb7d68ce6f21334"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.745518 19108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa97c35a63db4d6093b2d80fb2be432c"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "ecd7b5ace43a211d8c9d0feec6c88d37"
server_key_iv: "abd253dfaddb8645ccb7d68ce6f21334"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.750077 19108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:44.753189 19122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:44.754812 19108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:56:44.754963 19108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "fa97c35a63db4d6093b2d80fb2be432c"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "ecd7b5ace43a211d8c9d0feec6c88d37"
server_key_iv: "abd253dfaddb8645ccb7d68ce6f21334"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.755106 19108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:44.780620 19108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:44.781530 19108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:44.781718 19108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:44.782416 19108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:44.783654 19108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:44.783708 19108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:44.783759 19108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:44.783792 19108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:44.794658 19108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36275
I20260927 00:56:44.794697 19235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36275 every 8 connection(s)
I20260927 00:56:44.795864 19108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:44.796581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19108
I20260927 00:56:44.796751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:56:44.797093 392 external_mini_cluster.cc:1428] Setting key c6fd9f86ce100b37a6b725c4ece2a71d
I20260927 00:56:44.799381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35239
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:56:44.809088 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.798349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36271 (local address 127.0.98.62:35239)
0927 00:56:44.799334 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:44.799339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:44.800317 (+ 978us) server_negotiation.cc:407] Connection header received
0927 00:56:44.802065 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:44.802070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:44.802157 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:44.802275 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:44.803704 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.804421 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.806327 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.806621 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.806746 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.808346 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:44.808350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:44.808352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:44.808371 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:44.808399 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:44.808408 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:44.808487 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:44.808748 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 00:56:44.808929 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857}
I20260927 00:56:44.810154 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:44.798984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35239 (local address 127.0.98.1:36271)
0927 00:56:44.800101 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:44.800185 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:44.801783 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:44.802567 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:44.802579 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:44.802707 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:44.803478 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:44.803498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.804589 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.804593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:44.806124 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:44.806140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:44.806831 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:44.806837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:44.807086 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:44.807925 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:44.807928 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:44.808005 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:44.808112 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:44.808508 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:44.808519 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:44.808526 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:56:44.808674 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 00:56:44.809034 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":214,"threads_started":1}
I20260927 00:56:44.811746 19236 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35239
I20260927 00:56:44.812132 19236 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:44.812914 19236 heartbeater.cc:511] Master 127.0.98.62:35239 requested a full tablet report, sending...
I20260927 00:56:44.814939 19048 ts_manager.cc:194] Registered new tserver with Master: fa97c35a63db4d6093b2d80fb2be432c (127.0.98.1:36275)
I20260927 00:56:44.816361 19048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36271
W20260927 00:56:44.920012 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:44.920292 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:44.920346 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:44.924355 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:44.924414 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:44.924487 19239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:44.929332 19239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:44.930440 19239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:44.932016 19239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:44.942193 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:44.942227 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:44.942224 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:44.942478 19239 server_base.cc:1034] running on GCE node
I20260927 00:56:44.943261 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:44.944079 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:44.945267 19239 hybrid_clock.cc:648] HybridClock initialized: now 1790470604945247 us; error 38 us; skew 500 ppm
I20260927 00:56:44.947826 19239 webserver.cc:460] Webserver started at http://127.0.98.2:40703/ using document root <none> and password file <none>
I20260927 00:56:44.948535 19239 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:44.948607 19239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:44.948853 19239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:44.951524 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "af7f4822288e419fa38af8e1567a2b8a"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "5283ae7a058cd179257cfbae6e4f1f54"
server_key_iv: "ebed3660fda90623d3aa0f29b3b72356"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.952168 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af7f4822288e419fa38af8e1567a2b8a"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "5283ae7a058cd179257cfbae6e4f1f54"
server_key_iv: "ebed3660fda90623d3aa0f29b3b72356"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.956962 19239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:56:44.960729 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:44.962497 19239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:44.962662 19239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "af7f4822288e419fa38af8e1567a2b8a"
format_stamp: "Formatted at 2026-09-27 00:56:44 on dist-test-slave-9fh2"
server_key: "5283ae7a058cd179257cfbae6e4f1f54"
server_key_iv: "ebed3660fda90623d3aa0f29b3b72356"
server_key_version: "encryptionkey@0"
I20260927 00:56:44.962813 19239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:44.990097 19239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:44.991179 19239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:44.991408 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:44.992197 19239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:44.993363 19239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:44.993413 19239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:44.993460 19239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:44.993492 19239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:45.004284 19239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41409
I20260927 00:56:45.004379 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41409 every 8 connection(s)
I20260927 00:56:45.005437 19239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:45.007498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19239
I20260927 00:56:45.007679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:56:45.008008 392 external_mini_cluster.cc:1428] Setting key 78a984502fa6fb530f56d1844465357e
I20260927 00:56:45.010541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35239
--builtin_ntp_servers=127.0.98.20:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:56:45.017750 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.007791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34507 (local address 127.0.98.62:35239)
0927 00:56:45.007997 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.008004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.010217 (+ 2213us) server_negotiation.cc:407] Connection header received
0927 00:56:45.011824 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.011829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.011916 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.012061 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:45.013592 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.014154 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.015695 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.015927 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.016047 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.017245 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:45.017248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:45.017250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:45.017265 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:45.017284 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:45.017292 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:45.017350 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.017510 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.017585 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:56:45.018658 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.008444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35239 (local address 127.0.98.2:34507)
0927 00:56:45.010014 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.010074 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.011534 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.012369 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.012381 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.012518 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:45.013300 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.013322 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.014294 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.014297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.015478 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.015492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.016143 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.016149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.016318 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.016952 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:45.016954 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:45.016998 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:45.017077 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:45.017361 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:45.017371 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:45.017377 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.017505 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.017816 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":192,"threads_started":1}
I20260927 00:56:45.020010 19367 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35239
I20260927 00:56:45.020260 19367 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:45.020817 19367 heartbeater.cc:511] Master 127.0.98.62:35239 requested a full tablet report, sending...
I20260927 00:56:45.021909 19048 ts_manager.cc:194] Registered new tserver with Master: af7f4822288e419fa38af8e1567a2b8a (127.0.98.2:41409)
I20260927 00:56:45.022506 19048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34507
W20260927 00:56:45.147321 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:45.147680 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:45.147738 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:45.152307 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:45.152400 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:45.152498 19370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:45.158197 19370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:45.159828 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:45.161664 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:45.173362 19376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:45.173347 19378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:45.173352 19375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:45.173596 19370 server_base.cc:1034] running on GCE node
I20260927 00:56:45.174414 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:45.175374 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:45.176578 19370 hybrid_clock.cc:648] HybridClock initialized: now 1790470605176556 us; error 40 us; skew 500 ppm
I20260927 00:56:45.179019 19370 webserver.cc:460] Webserver started at http://127.0.98.3:38531/ using document root <none> and password file <none>
I20260927 00:56:45.179840 19370 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:45.179921 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:45.180133 19370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:45.182596 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f8a5b736999400f87850bdf8fa7a02b"
format_stamp: "Formatted at 2026-09-27 00:56:45 on dist-test-slave-9fh2"
server_key: "f6aef6627c9fc76a9e47574c03f34783"
server_key_iv: "89e70110dc67216b69d438d3f5358596"
server_key_version: "encryptionkey@0"
I20260927 00:56:45.183231 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f8a5b736999400f87850bdf8fa7a02b"
format_stamp: "Formatted at 2026-09-27 00:56:45 on dist-test-slave-9fh2"
server_key: "f6aef6627c9fc76a9e47574c03f34783"
server_key_iv: "89e70110dc67216b69d438d3f5358596"
server_key_version: "encryptionkey@0"
I20260927 00:56:45.188050 19370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:56:45.191358 19384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:45.193156 19370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:45.193316 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "9f8a5b736999400f87850bdf8fa7a02b"
format_stamp: "Formatted at 2026-09-27 00:56:45 on dist-test-slave-9fh2"
server_key: "f6aef6627c9fc76a9e47574c03f34783"
server_key_iv: "89e70110dc67216b69d438d3f5358596"
server_key_version: "encryptionkey@0"
I20260927 00:56:45.193467 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:45.238353 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:45.239468 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:45.239714 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:45.240579 19370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:45.241896 19370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:45.241964 19370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:45.242038 19370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:45.242069 19370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:45.254444 19370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42499
I20260927 00:56:45.254487 19497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42499 every 8 connection(s)
I20260927 00:56:45.255712 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:45.260612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19370
I20260927 00:56:45.260753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:56:45.261194 392 external_mini_cluster.cc:1428] Setting key dc84dc4856b5ed40b46d7d6629d96da9
I20260927 00:56:45.267639 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.258183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55847 (local address 127.0.98.62:35239)
0927 00:56:45.258352 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.258357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.260088 (+ 1731us) server_negotiation.cc:407] Connection header received
0927 00:56:45.261690 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.261694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.261854 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.261991 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:45.263287 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.264018 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.265336 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.265597 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.265706 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.266974 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:45.266978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:45.266980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:45.266998 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:45.267023 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:45.267032 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:45.267104 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.267396 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.267477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:56:45.268512 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.258731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35239 (local address 127.0.98.3:55847)
0927 00:56:45.259851 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.259906 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.261425 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.262219 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.262229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.262355 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:45.263082 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.263103 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.264156 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.264160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.265157 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.265171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.265764 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.265767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.265928 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.266648 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:45.266650 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:45.266697 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:45.266778 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:45.267131 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:45.267139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:45.267185 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.267341 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.267643 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":212,"threads_started":1}
I20260927 00:56:45.270092 19498 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35239
I20260927 00:56:45.270367 19498 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:45.271003 19498 heartbeater.cc:511] Master 127.0.98.62:35239 requested a full tablet report, sending...
I20260927 00:56:45.272503 19048 ts_manager.cc:194] Registered new tserver with Master: 9f8a5b736999400f87850bdf8fa7a02b (127.0.98.3:42499)
I20260927 00:56:45.273288 19048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55847
I20260927 00:56:45.276434 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:45.283293 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.279085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.0.98.62:35239)
0927 00:56:45.279307 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.279313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.279435 (+ 122us) server_negotiation.cc:407] Connection header received
0927 00:56:45.279585 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.279588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.279638 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.279709 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:45.280032 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.280594 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.281501 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.281733 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.281832 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.282744 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:45.282748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:45.282750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:45.282763 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:45.282790 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:45.282798 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:45.282892 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.283041 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.283089 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 00:56:45.287029 19048 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:45.290306 19048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:45.307507 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.301235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42499 (local address 127.0.0.1:38802)
0927 00:56:45.301884 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.301941 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.302126 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.302849 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.302855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.302891 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:45.303107 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.303117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.304661 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.304665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.305756 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.305764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.306773 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.306777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.306918 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.306975 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.307101 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.307291 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":5,"spinlock_wait_cycles":4992,"thread_start_us":149,"threads_started":1}
I20260927 00:56:45.307572 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.301363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.0.98.1:36275)
0927 00:56:45.301786 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.301793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.302202 (+ 409us) server_negotiation.cc:407] Connection header received
0927 00:56:45.302392 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.302397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.302569 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.302687 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:45.303308 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.304191 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.305795 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.306605 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.306688 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.306898 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.307273 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.307386 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":125,"threads_started":1}
I20260927 00:56:45.307653 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.302424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.98.2:41409)
0927 00:56:45.302884 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.302898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.303280 (+ 382us) server_negotiation.cc:407] Connection header received
0927 00:56:45.303382 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.303388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.303612 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.303786 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:45.304034 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.304988 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.306318 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.306985 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.307097 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.307333 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.307390 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.307495 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":117,"threads_started":1}
I20260927 00:56:45.308112 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.301356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.3:42499)
0927 00:56:45.301963 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.301974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.302262 (+ 288us) server_negotiation.cc:407] Connection header received
0927 00:56:45.302345 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.302350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.302563 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.302694 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:45.303275 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.304536 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.305892 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.306644 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.306744 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.306898 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.307854 (+ 956us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.307973 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":116,"threads_started":1}
I20260927 00:56:45.307567 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.302196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41409 (local address 127.0.0.1:34094)
0927 00:56:45.303027 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.303039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.303180 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.303794 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.303798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.303820 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:45.303904 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.303909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.305103 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.305107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.306194 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.306205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.307109 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.307112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.307236 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.307260 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.307315 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.307404 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260927 00:56:45.308020 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.301235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36275 (local address 127.0.0.1:45482)
0927 00:56:45.302105 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.302117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.302264 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.302850 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.302855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.302890 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:45.303214 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.303222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.304286 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.304289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.305651 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.305662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.306996 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.306998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.307083 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.307112 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.307814 (+ 702us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.307895 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":591,"spinlock_wait_cycles":16256,"thread_start_us":140,"threads_started":1}
I20260927 00:56:45.311398 19301 tablet_service.cc:1467] Processing CreateTablet for tablet d3ea443989d943789dac1650cf188924 (DEFAULT_TABLE table=test-table [id=59b15a3ac1e648dcb117507d95a632bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:45.311615 19432 tablet_service.cc:1467] Processing CreateTablet for tablet d3ea443989d943789dac1650cf188924 (DEFAULT_TABLE table=test-table [id=59b15a3ac1e648dcb117507d95a632bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:45.312475 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ea443989d943789dac1650cf188924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:45.312767 19432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ea443989d943789dac1650cf188924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:45.318997 19515 tablet_bootstrap.cc:492] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: Bootstrap starting.
I20260927 00:56:45.320602 19516 tablet_bootstrap.cc:492] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Bootstrap starting.
I20260927 00:56:45.321532 19170 tablet_service.cc:1467] Processing CreateTablet for tablet d3ea443989d943789dac1650cf188924 (DEFAULT_TABLE table=test-table [id=59b15a3ac1e648dcb117507d95a632bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:45.322543 19515 tablet_bootstrap.cc:654] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:45.322690 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ea443989d943789dac1650cf188924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:45.322811 19516 tablet_bootstrap.cc:654] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:45.323562 19515 log.cc:824] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:45.323683 19516 log.cc:824] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:45.326062 19515 tablet_bootstrap.cc:492] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: No bootstrap required, opened a new log
I20260927 00:56:45.326061 19516 tablet_bootstrap.cc:492] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: No bootstrap required, opened a new log
I20260927 00:56:45.326299 19516 ts_tablet_manager.cc:1397] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:56:45.326299 19515 ts_tablet_manager.cc:1397] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:56:45.329434 19519 tablet_bootstrap.cc:492] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: Bootstrap starting.
I20260927 00:56:45.331276 19516 raft_consensus.cc:348] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.331276 19515 raft_consensus.cc:348] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.331624 19519 tablet_bootstrap.cc:654] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:45.331634 19515 raft_consensus.cc:374] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:45.331634 19516 raft_consensus.cc:374] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:45.331691 19515 raft_consensus.cc:729] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7f4822288e419fa38af8e1567a2b8a, State: Initialized, Role: FOLLOWER
I20260927 00:56:45.331691 19516 raft_consensus.cc:729] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8a5b736999400f87850bdf8fa7a02b, State: Initialized, Role: FOLLOWER
I20260927 00:56:45.332242 19516 consensus_queue.cc:260] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.332289 19515 consensus_queue.cc:260] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.333138 19519 log.cc:824] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:45.333482 19516 ts_tablet_manager.cc:1428] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:56:45.333482 19515 ts_tablet_manager.cc:1428] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:56:45.333951 19498 heartbeater.cc:503] Master 127.0.98.62:35239 was elected leader, sending a full tablet report...
I20260927 00:56:45.334048 19367 heartbeater.cc:503] Master 127.0.98.62:35239 was elected leader, sending a full tablet report...
I20260927 00:56:45.335772 19519 tablet_bootstrap.cc:492] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: No bootstrap required, opened a new log
I20260927 00:56:45.336031 19519 ts_tablet_manager.cc:1397] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:56:45.340644 19519 raft_consensus.cc:348] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.340962 19519 raft_consensus.cc:374] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:45.341023 19519 raft_consensus.cc:729] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa97c35a63db4d6093b2d80fb2be432c, State: Initialized, Role: FOLLOWER
I20260927 00:56:45.341564 19519 consensus_queue.cc:260] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.342439 19236 heartbeater.cc:503] Master 127.0.98.62:35239 was elected leader, sending a full tablet report...
I20260927 00:56:45.342767 19519 ts_tablet_manager.cc:1428] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:56:45.347544 19520 raft_consensus.cc:482] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:45.347776 19520 raft_consensus.cc:504] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.349013 19520 leader_election.cc:288] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa97c35a63db4d6093b2d80fb2be432c (127.0.98.1:36275), 9f8a5b736999400f87850bdf8fa7a02b (127.0.98.3:42499)
I20260927 00:56:45.353744 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.349293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47377 (local address 127.0.98.1:36275)
0927 00:56:45.349409 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.349412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.349506 (+ 94us) server_negotiation.cc:407] Connection header received
0927 00:56:45.349791 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.349809 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.349885 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.349990 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:45.350708 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.351287 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.352779 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.353199 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.353295 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.353356 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.353535 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.353604 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:56:45.353780 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.349226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36275 (local address 127.0.98.2:47377)
0927 00:56:45.349396 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.349417 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.349562 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.350399 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.350402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.350424 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:45.350579 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.350587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.351421 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.351425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.352660 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.352672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.353316 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.353319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.353424 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.353447 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.353513 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.353593 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":29}
I20260927 00:56:45.353826 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.349599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59757 (local address 127.0.98.3:42499)
0927 00:56:45.349716 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.349720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.350028 (+ 308us) server_negotiation.cc:407] Connection header received
0927 00:56:45.350115 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.350117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.350167 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.350241 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:45.350525 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.351270 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.352792 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.353216 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.353318 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.353399 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.353574 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.353659 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:56:45.354228 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.349423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42499 (local address 127.0.98.2:59757)
0927 00:56:45.349876 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.349893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:45.350022 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:45.350263 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:45.350265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:45.350280 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:45.350399 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.350405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.351422 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.351425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.352664 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:45.352676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.353356 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.353360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:45.353457 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.353478 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:56:45.353548 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:56:45.353639 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":121,"thread_start_us":197,"threads_started":1}
I20260927 00:56:45.354400 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ea443989d943789dac1650cf188924" candidate_uuid: "af7f4822288e419fa38af8e1567a2b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa97c35a63db4d6093b2d80fb2be432c" is_pre_election: true
I20260927 00:56:45.354764 19190 raft_consensus.cc:2454] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7f4822288e419fa38af8e1567a2b8a in term 0.
I20260927 00:56:45.355032 19452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ea443989d943789dac1650cf188924" candidate_uuid: "af7f4822288e419fa38af8e1567a2b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8a5b736999400f87850bdf8fa7a02b" is_pre_election: true
I20260927 00:56:45.355417 19452 raft_consensus.cc:2454] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7f4822288e419fa38af8e1567a2b8a in term 0.
I20260927 00:56:45.355351 19256 leader_election.cc:302] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af7f4822288e419fa38af8e1567a2b8a, fa97c35a63db4d6093b2d80fb2be432c; no voters:
I20260927 00:56:45.355793 19520 raft_consensus.cc:2781] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:45.355911 19520 raft_consensus.cc:482] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:45.355965 19520 raft_consensus.cc:3037] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:45.357025 19520 raft_consensus.cc:504] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.357395 19520 leader_election.cc:288] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [CANDIDATE]: Term 1 election: Requested vote from peers fa97c35a63db4d6093b2d80fb2be432c (127.0.98.1:36275), 9f8a5b736999400f87850bdf8fa7a02b (127.0.98.3:42499)
I20260927 00:56:45.357874 19452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ea443989d943789dac1650cf188924" candidate_uuid: "af7f4822288e419fa38af8e1567a2b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8a5b736999400f87850bdf8fa7a02b"
I20260927 00:56:45.357895 19190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ea443989d943789dac1650cf188924" candidate_uuid: "af7f4822288e419fa38af8e1567a2b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa97c35a63db4d6093b2d80fb2be432c"
I20260927 00:56:45.358042 19452 raft_consensus.cc:3037] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:45.358079 19190 raft_consensus.cc:3037] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:45.359727 19452 raft_consensus.cc:2454] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7f4822288e419fa38af8e1567a2b8a in term 1.
I20260927 00:56:45.359727 19190 raft_consensus.cc:2454] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7f4822288e419fa38af8e1567a2b8a in term 1.
I20260927 00:56:45.360070 19256 leader_election.cc:302] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af7f4822288e419fa38af8e1567a2b8a, fa97c35a63db4d6093b2d80fb2be432c; no voters:
I20260927 00:56:45.360260 19520 raft_consensus.cc:2781] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:45.360517 19520 raft_consensus.cc:686] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 1 LEADER]: Becoming Leader. State: Replica: af7f4822288e419fa38af8e1567a2b8a, State: Running, Role: LEADER
I20260927 00:56:45.360962 19520 consensus_queue.cc:237] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } }
I20260927 00:56:45.365154 19048 catalog_manager.cc:5495] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a reported cstate change: term changed from 0 to 1, leader changed from <none> to af7f4822288e419fa38af8e1567a2b8a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "af7f4822288e419fa38af8e1567a2b8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7f4822288e419fa38af8e1567a2b8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41409 } health_report { overall_health: HEALTHY } } }
I20260927 00:56:45.390938 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.386594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.98.2:41409)
0927 00:56:45.386802 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:45.386806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:45.386827 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:56:45.386930 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:45.386933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:45.387009 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:45.387207 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:45.387503 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.388107 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.389065 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:45.389374 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:45.389472 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:45.389863 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:45.390443 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:45.390578 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:45.390708 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 00:56:45.390756 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:56:45.398290 19452 raft_consensus.cc:1261] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 1 FOLLOWER]: Refusing update from remote peer af7f4822288e419fa38af8e1567a2b8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:45.398290 19190 raft_consensus.cc:1261] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 1 FOLLOWER]: Refusing update from remote peer af7f4822288e419fa38af8e1567a2b8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:45.399237 19520 consensus_queue.cc:1035] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f8a5b736999400f87850bdf8fa7a02b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:45.399539 19525 consensus_queue.cc:1035] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa97c35a63db4d6093b2d80fb2be432c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:45.408733 19530 mvcc.cc:205] Tried to move back new op lower bound from 7333767599701704704 to 7333767599561961472. Current Snapshot: MvccSnapshot[applied={T|T < 7333767599701704704}]
I20260927 00:56:45.420639 19046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:45.420858 19046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:45.423928 19046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778304831a31453392756edbb4381e33: Sending DeleteTablet for 3 replicas of tablet d3ea443989d943789dac1650cf188924
I20260927 00:56:45.424948 19170 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ea443989d943789dac1650cf188924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:45 UTC) from {username='slave'} at 127.0.0.1:45482
I20260927 00:56:45.425107 19301 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ea443989d943789dac1650cf188924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:45 UTC) from {username='slave'} at 127.0.0.1:34094
I20260927 00:56:45.425107 19432 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ea443989d943789dac1650cf188924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:45 UTC) from {username='slave'} at 127.0.0.1:38802
I20260927 00:56:45.425875 19539 tablet_replica.cc:330] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c: stopping tablet replica
I20260927 00:56:45.426177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19108
I20260927 00:56:45.426340 19539 raft_consensus.cc:2229] T d3ea443989d943789dac1650cf188924 P fa97c35a63db4d6093b2d80fb2be432c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:45.427713 19541 tablet_replica.cc:330] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: stopping tablet replica
I20260927 00:56:45.428097 19541 raft_consensus.cc:2229] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:45.428467 19541 raft_consensus.cc:2258] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:45.428923 19540 tablet_replica.cc:330] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a: stopping tablet replica
I20260927 00:56:45.429207 19540 raft_consensus.cc:2229] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:56:45.429687 19540 raft_consensus.cc:2258] T d3ea443989d943789dac1650cf188924 P af7f4822288e419fa38af8e1567a2b8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:45.431527 19541 ts_tablet_manager.cc:1905] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:45.434439 19541 ts_tablet_manager.cc:1918] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:45.434564 19541 log.cc:1196] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/d3ea443989d943789dac1650cf188924
I20260927 00:56:45.434873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19239
W20260927 00:56:45.436067 19034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36275 (error 108)
I20260927 00:56:45.436528 19541 ts_tablet_manager.cc:1939] T d3ea443989d943789dac1650cf188924 P 9f8a5b736999400f87850bdf8fa7a02b: Deleting consensus metadata
I20260927 00:56:45.437676 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.437368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36275 (local address 127.0.0.1:45490)
0927 00:56:45.437515 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.437590 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:56:45.437999 19034 catalog_manager.cc:4584] TS fa97c35a63db4d6093b2d80fb2be432c (127.0.98.1:36275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3ea443989d943789dac1650cf188924: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36275: connect: Connection refused (error 111)
I20260927 00:56:45.438261 19035 catalog_manager.cc:4841] TS 9f8a5b736999400f87850bdf8fa7a02b (127.0.98.3:42499): tablet d3ea443989d943789dac1650cf188924 (table test-table [id=59b15a3ac1e648dcb117507d95a632bd]) successfully deleted
I20260927 00:56:45.442199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19370
I20260927 00:56:45.442564 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:45.442192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41409 (local address 127.0.0.1:34114)
0927 00:56:45.442377 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:45.442442 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:56:45.448485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19017
2026-09-27T00:56:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:45 dist-test-slave-9fh2 krb5kdc[19545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:45 dist-test-slave-9fh2 krb5kdc[19545](info): set up 2 sockets
Sep 27 00:56:45 dist-test-slave-9fh2 krb5kdc[19545](info): commencing operation
krb5kdc: starting...
W20260927 00:56:46.427639 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:46 dist-test-slave-9fh2 krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:56:46.573997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:46.706017 19558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:46.706415 19558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:46.706482 19558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:46.711031 19558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:46.711109 19558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:46.711138 19558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:46.711184 19558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:46.711213 19558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:46.717303 19558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:46.718804 19558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:46.720230 19558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:46.730145 19564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:46.730271 19558 server_base.cc:1034] running on GCE node
W20260927 00:56:46.730145 19563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:46.730151 19566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:46.731423 19558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:46.732724 19558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:46.733903 19558 hybrid_clock.cc:648] HybridClock initialized: now 1790470606733875 us; error 51 us; skew 500 ppm
Sep 27 00:56:46 dist-test-slave-9fh2 krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:46.740991 19558 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:56:46.742030 19558 webserver.cc:460] Webserver started at http://127.0.98.62:40805/ using document root <none> and password file <none>
I20260927 00:56:46.742663 19558 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:46.742735 19558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:46.742952 19558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:46.745193 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "5d226a5ff7394d48872b96bd5b1bfe08"
format_stamp: "Formatted at 2026-09-27 00:56:46 on dist-test-slave-9fh2"
server_key: "d49c188c78a597a96d71218398690209"
server_key_iv: "07b7b8cf7da4f91aaf3f98f1f76bc691"
server_key_version: "encryptionkey@0"
I20260927 00:56:46.745797 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d226a5ff7394d48872b96bd5b1bfe08"
format_stamp: "Formatted at 2026-09-27 00:56:46 on dist-test-slave-9fh2"
server_key: "d49c188c78a597a96d71218398690209"
server_key_iv: "07b7b8cf7da4f91aaf3f98f1f76bc691"
server_key_version: "encryptionkey@0"
I20260927 00:56:46.750205 19558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:46.752939 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:46.754499 19558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:56:46.754632 19558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "5d226a5ff7394d48872b96bd5b1bfe08"
format_stamp: "Formatted at 2026-09-27 00:56:46 on dist-test-slave-9fh2"
server_key: "d49c188c78a597a96d71218398690209"
server_key_iv: "07b7b8cf7da4f91aaf3f98f1f76bc691"
server_key_version: "encryptionkey@0"
I20260927 00:56:46.754743 19558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:46.767741 19558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:46.770933 19558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:46.771119 19558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:46.778582 19558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37087
I20260927 00:56:46.778674 19625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37087 every 8 connection(s)
I20260927 00:56:46.779848 19558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:46.782218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19558
I20260927 00:56:46.782387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:46.782739 392 external_mini_cluster.cc:1428] Setting key feb632a6528fbd83475b0ba9b2432823
I20260927 00:56:46.783805 19626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:46.791867 19626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08: Bootstrap starting.
Sep 27 00:56:46 dist-test-slave-9fh2 krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:46.794878 19626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:46.795773 19626 log.cc:824] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:46.798102 19626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08: No bootstrap required, opened a new log
I20260927 00:56:46.798192 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:46.784381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.98.62:37087)
0927 00:56:46.784851 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:46.784867 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:46.784907 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:56:46.785586 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:46.785604 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:46.785959 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:46.786256 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:46.787301 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:46.788329 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:46.789574 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:46.790084 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:46.790212 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:46.793438 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:46.793443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:46.793459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:46.793490 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:46.795351 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:46.795794 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:46.795800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:46.795804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:46.795926 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:46.796164 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:46.796167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:46.796169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:46.796583 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:46.796795 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:46.797070 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:56:46.797390 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":130,"threads_started":1}
I20260927 00:56:46.802418 19626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260927 00:56:46.802687 19626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:46.802742 19626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d226a5ff7394d48872b96bd5b1bfe08, State: Initialized, Role: FOLLOWER
I20260927 00:56:46.803247 19626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260927 00:56:46.803403 19626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:46.803471 19626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:46.803573 19626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:46.804797 19626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260927 00:56:46.805215 19626 leader_election.cc:302] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d226a5ff7394d48872b96bd5b1bfe08; no voters:
I20260927 00:56:46.805573 19626 leader_election.cc:288] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:46.805755 19631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:46.806013 19631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [term 1 LEADER]: Becoming Leader. State: Replica: 5d226a5ff7394d48872b96bd5b1bfe08, State: Running, Role: LEADER
I20260927 00:56:46.806380 19631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260927 00:56:46.806998 19626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:46.808228 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d226a5ff7394d48872b96bd5b1bfe08. Latest consensus state: current_term: 1 leader_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } } }
I20260927 00:56:46.808384 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:46.808678 19632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d226a5ff7394d48872b96bd5b1bfe08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } } }
I20260927 00:56:46.808771 19632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:46.808911 19640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:46.812583 19640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:46.820261 19640 catalog_manager.cc:1347] Generated new cluster ID: 7d6485774c8b4a6b933761c4dd8ad0ba
I20260927 00:56:46.820338 19640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:46.830094 19640 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:46.831758 19640 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:46.838506 19640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08: Generated new TSK 0
I20260927 00:56:46.839738 19640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:56:46.874567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37087
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:46.992156 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:46.992457 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:46.992522 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:46.996610 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:46.996685 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:46.996768 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:47.002275 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:47.003477 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:47.004541 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:47.015532 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:47.015552 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:47.015657 19652 server_base.cc:1034] running on GCE node
W20260927 00:56:47.015552 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:47.016395 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:47.017303 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:47.018523 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790470607018474 us; error 37 us; skew 500 ppm
Sep 27 00:56:47 dist-test-slave-9fh2 krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:47.022112 19652 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:56:47.023556 19652 webserver.cc:460] Webserver started at http://127.0.98.1:38511/ using document root <none> and password file <none>
I20260927 00:56:47.024259 19652 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:47.024334 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:47.024572 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:47.027088 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "275074518afd4b9689d8444db366df6f"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "9d653bfa63d24dddabab58d1fc9f2e7e"
server_key_iv: "e2b5bc1e4707bd5bdddfa1b3a3d7ae0b"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.027796 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "275074518afd4b9689d8444db366df6f"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "9d653bfa63d24dddabab58d1fc9f2e7e"
server_key_iv: "e2b5bc1e4707bd5bdddfa1b3a3d7ae0b"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.032548 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:56:47.035759 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.037401 19652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:47.037567 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "275074518afd4b9689d8444db366df6f"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "9d653bfa63d24dddabab58d1fc9f2e7e"
server_key_iv: "e2b5bc1e4707bd5bdddfa1b3a3d7ae0b"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.037699 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:47.066082 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:47.069989 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:47.070206 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:47.070945 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:47.072009 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:47.072062 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.072122 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:47.072146 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.082623 19652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38561
I20260927 00:56:47.082645 19780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38561 every 8 connection(s)
I20260927 00:56:47.083825 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:47.093362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19652
I20260927 00:56:47.093516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:56:47.093824 392 external_mini_cluster.cc:1428] Setting key b74f11d049f867f7818172fbd6b50454
Sep 27 00:56:47 dist-test-slave-9fh2 krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:47.099676 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.086130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60035 (local address 127.0.98.62:37087)
0927 00:56:47.086346 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.086353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.087517 (+ 1164us) server_negotiation.cc:407] Connection header received
0927 00:56:47.088462 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.088466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.088533 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.088651 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:47.090009 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.090667 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.091965 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.092245 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.092341 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.095247 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:47.095252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:47.095255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:47.095282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:47.096907 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.097431 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.097434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.097435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.097538 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.098084 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.098088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.098090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.098355 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:47.098465 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.099341 (+ 876us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.099495 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:56:47.100889 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.086683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37087 (local address 127.0.98.1:60035)
0927 00:56:47.087326 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.087378 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.088235 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.088875 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.088883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.089287 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:47.089814 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.089827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.090818 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.090820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.091809 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.091820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.092422 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.092425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.092583 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.093282 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:47.093298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:47.095013 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:47.097042 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:47.097048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:47.097065 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:47.097331 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:47.097703 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:47.097708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:47.097711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:47.097973 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:47.098515 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:47.098523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:47.098961 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.099275 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.099678 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":155,"threads_started":1}
I20260927 00:56:47.102576 19781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37087
I20260927 00:56:47.102955 19781 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:47.103737 19781 heartbeater.cc:511] Master 127.0.98.62:37087 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:56:47.105823 19590 ts_manager.cc:194] Registered new tserver with Master: 275074518afd4b9689d8444db366df6f (127.0.98.1:38561)
I20260927 00:56:47.107383 19590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:56:47.124558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37087
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:47.257128 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:47.257525 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:47.257579 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:47.262120 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:47.262224 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:47.262301 19786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:47.268710 19786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:47.270171 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:47.271531 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:47.283633 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:47.283624 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:47.283624 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:47.283921 19786 server_base.cc:1034] running on GCE node
I20260927 00:56:47.284432 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:47.285262 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:47.286487 19786 hybrid_clock.cc:648] HybridClock initialized: now 1790470607286445 us; error 56 us; skew 500 ppm
Sep 27 00:56:47 dist-test-slave-9fh2 krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:47.289968 19786 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:56:47.291203 19786 webserver.cc:460] Webserver started at http://127.0.98.2:38473/ using document root <none> and password file <none>
I20260927 00:56:47.291908 19786 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:47.291975 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:47.292191 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:47.294772 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "84eba8e8180646b28f479a7eed279d7b"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "3283781fd81088783edd858e633cadf2"
server_key_iv: "e6125dbcbcc2a9e68924e1bf7788ee8a"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.295454 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84eba8e8180646b28f479a7eed279d7b"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "3283781fd81088783edd858e633cadf2"
server_key_iv: "e6125dbcbcc2a9e68924e1bf7788ee8a"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.300349 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:56:47.303613 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.305425 19786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:56:47.305601 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "84eba8e8180646b28f479a7eed279d7b"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "3283781fd81088783edd858e633cadf2"
server_key_iv: "e6125dbcbcc2a9e68924e1bf7788ee8a"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.305747 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:47.325452 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:47.329566 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:47.329787 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:47.330547 19786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:47.331848 19786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:47.331911 19786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.331960 19786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:47.331984 19786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.343559 19786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39293
I20260927 00:56:47.343758 19914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39293 every 8 connection(s)
I20260927 00:56:47.344816 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:47.353026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19786
I20260927 00:56:47.353143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:56:47.353410 392 external_mini_cluster.cc:1428] Setting key 18a95235f23aa25214f7afa4491687d8
Sep 27 00:56:47 dist-test-slave-9fh2 krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:47.362344 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.347139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37277 (local address 127.0.98.62:37087)
0927 00:56:47.347379 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.347385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.348420 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 00:56:47.349996 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.350000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.350084 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.350189 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:47.351886 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.352443 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.353976 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.354270 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.354369 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.357753 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:47.357757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:47.357760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:47.357891 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:47.359597 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.360437 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.360440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.360442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.360530 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.360968 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.360970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.360972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.361189 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:47.361272 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.362035 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.362176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:56:47.363531 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.347554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37087 (local address 127.0.98.2:37277)
0927 00:56:47.348169 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.348228 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.349713 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.350480 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.350489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.350973 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:47.351699 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.351717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.352621 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.352624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.353836 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.353851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.354465 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.354469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.354709 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.355483 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:47.355503 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:47.357434 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:47.359761 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:47.359770 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:47.359792 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:47.360269 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:47.360639 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:47.360642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:47.360644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:47.360859 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:47.361302 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:47.361310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:47.361709 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.361943 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.362359 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":152,"threads_started":1}
I20260927 00:56:47.365056 19915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37087
I20260927 00:56:47.365288 19915 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:47.365826 19915 heartbeater.cc:511] Master 127.0.98.62:37087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:56:47.367205 19590 ts_manager.cc:194] Registered new tserver with Master: 84eba8e8180646b28f479a7eed279d7b (127.0.98.2:39293)
I20260927 00:56:47.367965 19590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:56:47.382501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37087
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:47.516449 19920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:47.516842 19920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:47.516913 19920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:47.521900 19920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:47.522002 19920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:47.522104 19920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:47.528469 19920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:47.529913 19920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:47.531190 19920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:47.542500 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:47.542567 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:47.542640 19920 server_base.cc:1034] running on GCE node
W20260927 00:56:47.542832 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:47.543382 19920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:47.544328 19920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:47.545544 19920 hybrid_clock.cc:648] HybridClock initialized: now 1790470607545518 us; error 41 us; skew 500 ppm
Sep 27 00:56:47 dist-test-slave-9fh2 krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:47.549837 19920 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:56:47.551337 19920 webserver.cc:460] Webserver started at http://127.0.98.3:40877/ using document root <none> and password file <none>
I20260927 00:56:47.552068 19920 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:47.552145 19920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:47.552394 19920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:47.554948 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f82c8ed0a4e40da905840128e40eb35"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "55c40e2ed876cb5d7aa12e50176b34ec"
server_key_iv: "fff36badb9eed0eaf20fe43ab7015a1f"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.555649 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f82c8ed0a4e40da905840128e40eb35"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "55c40e2ed876cb5d7aa12e50176b34ec"
server_key_iv: "fff36badb9eed0eaf20fe43ab7015a1f"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.560523 19920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:56:47.563711 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.565383 19920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:47.565531 19920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "6f82c8ed0a4e40da905840128e40eb35"
format_stamp: "Formatted at 2026-09-27 00:56:47 on dist-test-slave-9fh2"
server_key: "55c40e2ed876cb5d7aa12e50176b34ec"
server_key_iv: "fff36badb9eed0eaf20fe43ab7015a1f"
server_key_version: "encryptionkey@0"
I20260927 00:56:47.565671 19920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:47.584426 19920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:47.588163 19920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:47.588361 19920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:47.589073 19920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:47.590039 19920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:47.590088 19920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.590132 19920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:47.590147 19920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:47.600987 19920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33461
I20260927 00:56:47.601033 20048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33461 every 8 connection(s)
I20260927 00:56:47.602173 19920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:47.610625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 19920
I20260927 00:56:47.610780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:56:47.611106 392 external_mini_cluster.cc:1428] Setting key 7fee2404f25ce177508b047a3d411ec6
Sep 27 00:56:47 dist-test-slave-9fh2 krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:56:47.618266 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.604355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34907 (local address 127.0.98.62:37087)
0927 00:56:47.604484 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.604488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.605678 (+ 1190us) server_negotiation.cc:407] Connection header received
0927 00:56:47.606609 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.606612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.606683 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.606806 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:47.608292 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.609053 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.610297 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.610576 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.610681 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.614026 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:47.614030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:47.614032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:47.614062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:47.615702 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.616522 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.616525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.616526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.616619 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.617013 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.617017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.617019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.617228 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:47.617324 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.618004 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.618133 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:56:47.619534 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.604866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37087 (local address 127.0.98.3:34907)
0927 00:56:47.605457 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.605509 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.606360 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.607003 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.607013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.607448 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:47.608081 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.608098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.609199 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.609207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.610151 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.610161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.610771 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.610775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.611014 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.611767 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:56:47.611784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:47.613794 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:47.615932 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:47.615940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:47.615991 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:47.616384 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:47.616719 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:56:47.616721 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:56:47.616722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:56:47.616899 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:56:47.617332 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:47.617338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:47.617709 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.617928 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.618374 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":143,"threads_started":1}
I20260927 00:56:47.620966 20049 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37087
I20260927 00:56:47.621289 20049 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:47.621910 20049 heartbeater.cc:511] Master 127.0.98.62:37087 requested a full tablet report, sending...
I20260927 00:56:47.623294 19590 ts_manager.cc:194] Registered new tserver with Master: 6f82c8ed0a4e40da905840128e40eb35 (127.0.98.3:33461)
I20260927 00:56:47.623996 19590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34907
I20260927 00:56:47.625828 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:47.635481 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.627928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.98.62:37087)
0927 00:56:47.628097 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.628104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.628306 (+ 202us) server_negotiation.cc:407] Connection header received
0927 00:56:47.628431 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.628434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.628478 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.628550 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:47.629248 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.629839 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.630665 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.630875 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.630959 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.632394 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:47.632397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:47.632400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:47.632422 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:47.633887 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.634272 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.634275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.634276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.634345 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:47.634600 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:47.634603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:47.634604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:47.634822 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:47.634940 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.635217 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.635328 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:56:47.638965 19590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:47.642267 19590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:47.658571 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.652793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38561 (local address 127.0.0.1:53352)
0927 00:56:47.653987 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.653999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.654073 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.654670 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.654673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.654688 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:47.654787 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.654793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.655860 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.655864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.657034 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.657046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.658070 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.658073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.658196 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.658238 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.658341 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.658449 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"thread_start_us":129,"threads_started":1}
I20260927 00:56:47.659003 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.653619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.0.98.1:38561)
0927 00:56:47.654305 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.654311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.654331 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:56:47.654395 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.654399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.654540 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.654663 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:47.654900 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.655751 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.657175 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.657963 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.658055 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.658202 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.658772 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.658871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":134,"threads_started":1}
I20260927 00:56:47.659605 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.652531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39293 (local address 127.0.0.1:41450)
0927 00:56:47.653493 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.653548 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.653698 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.654449 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.654455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.654479 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:47.654654 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.654661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.656440 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.656444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.657707 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.657717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.659097 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.659100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.659238 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.659266 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.659340 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.659437 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":58,"thread_start_us":117,"threads_started":1}
I20260927 00:56:47.659802 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.652571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33461 (local address 127.0.0.1:36332)
0927 00:56:47.653945 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.653958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.654091 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.654805 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.654807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.654824 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:47.654964 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.654972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.656477 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.656481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.657616 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.657626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.659258 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.659262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.659371 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.659400 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.659476 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.659570 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260927 00:56:47.660146 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.653356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.0.98.2:39293)
0927 00:56:47.653842 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.653863 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.653892 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:56:47.653986 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.653992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.654252 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.654837 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:47.654989 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.656279 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.657842 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.658739 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.658861 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.659029 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.659846 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.659984 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":111,"threads_started":1}
I20260927 00:56:47.660213 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.652864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.0.98.3:33461)
0927 00:56:47.653429 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.653441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.654036 (+ 595us) server_negotiation.cc:407] Connection header received
0927 00:56:47.654232 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.654238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.654449 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.654575 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:47.655112 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.656289 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.657766 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.658739 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.658886 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.659028 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.659519 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.659654 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":129,"threads_started":1}
I20260927 00:56:47.663733 19983 tablet_service.cc:1467] Processing CreateTablet for tablet 936e25e72eb54de5bcb3a03c584230b2 (DEFAULT_TABLE table=test-table [id=b4d30db2431a43e389ce965c32227e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:47.663734 19849 tablet_service.cc:1467] Processing CreateTablet for tablet 936e25e72eb54de5bcb3a03c584230b2 (DEFAULT_TABLE table=test-table [id=b4d30db2431a43e389ce965c32227e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:47.663954 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 936e25e72eb54de5bcb3a03c584230b2 (DEFAULT_TABLE table=test-table [id=b4d30db2431a43e389ce965c32227e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:47.664994 19849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936e25e72eb54de5bcb3a03c584230b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:47.664994 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936e25e72eb54de5bcb3a03c584230b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:47.665205 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936e25e72eb54de5bcb3a03c584230b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:47.672899 20066 tablet_bootstrap.cc:492] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: Bootstrap starting.
I20260927 00:56:47.673882 20067 tablet_bootstrap.cc:492] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Bootstrap starting.
I20260927 00:56:47.674873 20068 tablet_bootstrap.cc:492] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Bootstrap starting.
I20260927 00:56:47.676115 20067 tablet_bootstrap.cc:654] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:47.676403 20066 tablet_bootstrap.cc:654] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:47.677196 20067 log.cc:824] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:47.677409 20066 log.cc:824] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:47.677577 20068 tablet_bootstrap.cc:654] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:47.678464 20068 log.cc:824] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:47.679920 20067 tablet_bootstrap.cc:492] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: No bootstrap required, opened a new log
I20260927 00:56:47.680172 20066 tablet_bootstrap.cc:492] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: No bootstrap required, opened a new log
I20260927 00:56:47.680173 20067 ts_tablet_manager.cc:1397] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:47.680354 20066 ts_tablet_manager.cc:1397] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:56:47.680919 20068 tablet_bootstrap.cc:492] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: No bootstrap required, opened a new log
I20260927 00:56:47.681155 20068 ts_tablet_manager.cc:1397] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:47.684186 20068 raft_consensus.cc:348] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.684437 20068 raft_consensus.cc:374] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:47.684489 20068 raft_consensus.cc:729] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84eba8e8180646b28f479a7eed279d7b, State: Initialized, Role: FOLLOWER
I20260927 00:56:47.684691 20067 raft_consensus.cc:348] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.684924 20068 consensus_queue.cc:260] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.685035 20067 raft_consensus.cc:374] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:47.685111 20067 raft_consensus.cc:729] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f82c8ed0a4e40da905840128e40eb35, State: Initialized, Role: FOLLOWER
I20260927 00:56:47.685073 20066 raft_consensus.cc:348] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.685313 20066 raft_consensus.cc:374] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:47.685369 20066 raft_consensus.cc:729] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275074518afd4b9689d8444db366df6f, State: Initialized, Role: FOLLOWER
I20260927 00:56:47.685669 20067 consensus_queue.cc:260] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.685770 19915 heartbeater.cc:503] Master 127.0.98.62:37087 was elected leader, sending a full tablet report...
I20260927 00:56:47.685755 20066 consensus_queue.cc:260] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.685997 20068 ts_tablet_manager.cc:1428] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:56:47.686781 20067 ts_tablet_manager.cc:1428] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:56:47.686781 20066 ts_tablet_manager.cc:1428] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:56:47.687212 20049 heartbeater.cc:503] Master 127.0.98.62:37087 was elected leader, sending a full tablet report...
I20260927 00:56:47.687212 19781 heartbeater.cc:503] Master 127.0.98.62:37087 was elected leader, sending a full tablet report...
I20260927 00:56:47.753551 20074 raft_consensus.cc:482] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:47.753927 20074 raft_consensus.cc:504] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.755492 20074 leader_election.cc:288] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84eba8e8180646b28f479a7eed279d7b (127.0.98.2:39293), 6f82c8ed0a4e40da905840128e40eb35 (127.0.98.3:33461)
I20260927 00:56:47.760393 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.755948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35307 (local address 127.0.98.2:39293)
0927 00:56:47.756145 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.756150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.756439 (+ 289us) server_negotiation.cc:407] Connection header received
0927 00:56:47.756676 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.756680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.756733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.756803 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:47.757106 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.757824 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.759326 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.759813 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.759915 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.759996 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.760159 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.760239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:56:47.760430 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.755925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56577 (local address 127.0.98.3:33461)
0927 00:56:47.756181 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.756186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.756559 (+ 373us) server_negotiation.cc:407] Connection header received
0927 00:56:47.756733 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.756736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.756889 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.756980 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:47.757228 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.757886 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.759356 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.759867 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.759969 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.760036 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.760222 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.760296 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 00:56:47.760490 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.755827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39293 (local address 127.0.98.1:35307)
0927 00:56:47.756324 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.756336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.756493 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.756827 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.756830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.756845 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:47.756973 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.756980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.757979 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.757983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.759191 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.759203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.759935 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.759938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.760037 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.760064 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.760134 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.760229 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":4,"thread_start_us":172,"threads_started":1}
I20260927 00:56:47.760677 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.755879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33461 (local address 127.0.98.1:56577)
0927 00:56:47.756463 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.756476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:47.756604 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:47.757003 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:47.757006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:47.757021 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:47.757116 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.757122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.758034 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.758038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.759209 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:47.759221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.759990 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.759993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:47.760095 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.760124 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:56:47.760189 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:56:47.760288 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":58,"thread_start_us":153,"threads_started":1}
I20260927 00:56:47.761190 19869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936e25e72eb54de5bcb3a03c584230b2" candidate_uuid: "275074518afd4b9689d8444db366df6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84eba8e8180646b28f479a7eed279d7b" is_pre_election: true
I20260927 00:56:47.761373 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936e25e72eb54de5bcb3a03c584230b2" candidate_uuid: "275074518afd4b9689d8444db366df6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f82c8ed0a4e40da905840128e40eb35" is_pre_election: true
I20260927 00:56:47.761628 20003 raft_consensus.cc:2454] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 275074518afd4b9689d8444db366df6f in term 0.
I20260927 00:56:47.761628 19869 raft_consensus.cc:2454] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 275074518afd4b9689d8444db366df6f in term 0.
I20260927 00:56:47.762161 19670 leader_election.cc:302] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275074518afd4b9689d8444db366df6f, 84eba8e8180646b28f479a7eed279d7b; no voters:
I20260927 00:56:47.762462 20074 raft_consensus.cc:2781] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:47.762557 20074 raft_consensus.cc:482] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:47.762599 20074 raft_consensus.cc:3037] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:47.763895 20074 raft_consensus.cc:504] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.764298 20074 leader_election.cc:288] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [CANDIDATE]: Term 1 election: Requested vote from peers 84eba8e8180646b28f479a7eed279d7b (127.0.98.2:39293), 6f82c8ed0a4e40da905840128e40eb35 (127.0.98.3:33461)
I20260927 00:56:47.764701 19869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936e25e72eb54de5bcb3a03c584230b2" candidate_uuid: "275074518afd4b9689d8444db366df6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84eba8e8180646b28f479a7eed279d7b"
I20260927 00:56:47.764770 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936e25e72eb54de5bcb3a03c584230b2" candidate_uuid: "275074518afd4b9689d8444db366df6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f82c8ed0a4e40da905840128e40eb35"
I20260927 00:56:47.764865 19869 raft_consensus.cc:3037] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:47.764897 20003 raft_consensus.cc:3037] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:47.766361 20003 raft_consensus.cc:2454] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 275074518afd4b9689d8444db366df6f in term 1.
I20260927 00:56:47.766361 19869 raft_consensus.cc:2454] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 275074518afd4b9689d8444db366df6f in term 1.
I20260927 00:56:47.766733 19670 leader_election.cc:302] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275074518afd4b9689d8444db366df6f, 84eba8e8180646b28f479a7eed279d7b; no voters:
I20260927 00:56:47.766956 20074 raft_consensus.cc:2781] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:47.767271 20074 raft_consensus.cc:686] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 1 LEADER]: Becoming Leader. State: Replica: 275074518afd4b9689d8444db366df6f, State: Running, Role: LEADER
I20260927 00:56:47.767659 20074 consensus_queue.cc:237] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } }
I20260927 00:56:47.771620 19589 catalog_manager.cc:5495] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 275074518afd4b9689d8444db366df6f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "275074518afd4b9689d8444db366df6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275074518afd4b9689d8444db366df6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38561 } health_report { overall_health: HEALTHY } } }
I20260927 00:56:47.791747 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.787137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.0.98.1:38561)
0927 00:56:47.787372 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:47.787378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:47.787397 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:56:47.787521 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:47.787524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:47.787618 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:47.787744 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:47.787989 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.788915 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.789818 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:47.790183 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:47.790284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:47.790564 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:47.791304 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:47.791439 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:47.791488 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 00:56:47.791568 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:56:47.799332 19869 raft_consensus.cc:1261] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 1 FOLLOWER]: Refusing update from remote peer 275074518afd4b9689d8444db366df6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:47.799372 20003 raft_consensus.cc:1261] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 1 FOLLOWER]: Refusing update from remote peer 275074518afd4b9689d8444db366df6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:47.800225 20074 consensus_queue.cc:1035] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f82c8ed0a4e40da905840128e40eb35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:47.800663 20077 consensus_queue.cc:1035] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84eba8e8180646b28f479a7eed279d7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:47.809494 20080 mvcc.cc:205] Tried to move back new op lower bound from 7333767609535348736 to 7333767609419321344. Current Snapshot: MvccSnapshot[applied={T|T < 7333767609535348736}]
I20260927 00:56:47.811348 20081 mvcc.cc:205] Tried to move back new op lower bound from 7333767609535348736 to 7333767609419321344. Current Snapshot: MvccSnapshot[applied={T|T < 7333767609535348736}]
I20260927 00:56:47.820276 19589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:47.820571 19589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:47.823732 19589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d226a5ff7394d48872b96bd5b1bfe08: Sending DeleteTablet for 3 replicas of tablet 936e25e72eb54de5bcb3a03c584230b2
I20260927 00:56:47.824616 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 936e25e72eb54de5bcb3a03c584230b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53352
I20260927 00:56:47.824622 19849 tablet_service.cc:1514] Processing DeleteTablet for tablet 936e25e72eb54de5bcb3a03c584230b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41450
I20260927 00:56:47.824929 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet 936e25e72eb54de5bcb3a03c584230b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36332
I20260927 00:56:47.825187 20089 tablet_replica.cc:330] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: stopping tablet replica
I20260927 00:56:47.825520 20089 raft_consensus.cc:2229] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:47.825682 20091 tablet_replica.cc:330] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: stopping tablet replica
I20260927 00:56:47.826002 20091 raft_consensus.cc:2229] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:47.826327 20091 raft_consensus.cc:2258] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:47.826408 20090 tablet_replica.cc:330] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f: stopping tablet replica
I20260927 00:56:47.826886 20090 raft_consensus.cc:2229] T 936e25e72eb54de5bcb3a03c584230b2 P 275074518afd4b9689d8444db366df6f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:56:47.826930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19652
I20260927 00:56:47.828893 20091 ts_tablet_manager.cc:1905] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:47.826120 20089 raft_consensus.cc:2258] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:47.830892 20089 ts_tablet_manager.cc:1905] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:47.832414 20091 ts_tablet_manager.cc:1918] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:47.832544 20091 log.cc:1196] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/936e25e72eb54de5bcb3a03c584230b2
I20260927 00:56:47.832896 20091 ts_tablet_manager.cc:1939] T 936e25e72eb54de5bcb3a03c584230b2 P 6f82c8ed0a4e40da905840128e40eb35: Deleting consensus metadata
W20260927 00:56:47.833509 19577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38561 (error 108)
I20260927 00:56:47.833865 20089 ts_tablet_manager.cc:1918] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:47.833974 20089 log.cc:1196] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/936e25e72eb54de5bcb3a03c584230b2
I20260927 00:56:47.834262 20089 ts_tablet_manager.cc:1939] T 936e25e72eb54de5bcb3a03c584230b2 P 84eba8e8180646b28f479a7eed279d7b: Deleting consensus metadata
I20260927 00:56:47.834415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19786
I20260927 00:56:47.835057 19574 catalog_manager.cc:4841] TS 6f82c8ed0a4e40da905840128e40eb35 (127.0.98.3:33461): tablet 936e25e72eb54de5bcb3a03c584230b2 (table test-table [id=b4d30db2431a43e389ce965c32227e45]) successfully deleted
I20260927 00:56:47.835494 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.835053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38561 (local address 127.0.0.1:53362)
0927 00:56:47.835270 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.835380 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 00:56:47.836261 19577 catalog_manager.cc:4584] TS 275074518afd4b9689d8444db366df6f (127.0.98.1:38561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 936e25e72eb54de5bcb3a03c584230b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38561: connect: Connection refused (error 111)
I20260927 00:56:47.840334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19920
I20260927 00:56:47.841624 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:47.841231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39293 (local address 127.0.0.1:41456)
0927 00:56:47.841450 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:47.841511 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 00:56:47.845964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 19558
2026-09-27T00:56:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2397 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T00:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:47Z Disabled control of system clock
I20260927 00:56:47.878664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38407
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:56:48.013645 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:48.014079 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:48.014120 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:48.019194 20096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:48.019318 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:48.019333 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:48.019352 20096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:48.019368 20096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:48.025211 20096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:48.026889 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:48.028716 20096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:48.039409 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.039588 20096 server_base.cc:1034] running on GCE node
W20260927 00:56:48.039793 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:48.039464 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.040752 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:48.042277 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:48.043555 20096 hybrid_clock.cc:648] HybridClock initialized: now 1790470608043521 us; error 52 us; skew 500 ppm
I20260927 00:56:48.046247 20096 webserver.cc:460] Webserver started at http://127.0.98.62:34955/ using document root <none> and password file <none>
I20260927 00:56:48.046999 20096 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:48.047078 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:48.047340 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:48.050330 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "a22977ca4726427d9d1cae6cc653de00"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "e40eed8d7b473fc737fee1bdf43591ed"
server_key_iv: "1d2aa991e4a6e6e57d0b48ac19ead0a7"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.051113 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "a22977ca4726427d9d1cae6cc653de00"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "e40eed8d7b473fc737fee1bdf43591ed"
server_key_iv: "1d2aa991e4a6e6e57d0b48ac19ead0a7"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.056592 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:56:48.060158 20110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.062171 20096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:56:48.062381 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "a22977ca4726427d9d1cae6cc653de00"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "e40eed8d7b473fc737fee1bdf43591ed"
server_key_iv: "1d2aa991e4a6e6e57d0b48ac19ead0a7"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.062541 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:48.101147 20096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:48.102227 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:48.102463 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:48.111526 20096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38407
I20260927 00:56:48.111555 20162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38407 every 8 connection(s)
I20260927 00:56:48.112869 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:48.116596 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:48.117930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20096
I20260927 00:56:48.118074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:48.118409 392 external_mini_cluster.cc:1428] Setting key ce24c7a7516d15ed1dd4cb97de1fbbc7
I20260927 00:56:48.124322 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: Bootstrap starting.
I20260927 00:56:48.127046 20163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:48.127986 20163 log.cc:824] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:48.130172 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: No bootstrap required, opened a new log
I20260927 00:56:48.130743 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.120374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.98.62:38407)
0927 00:56:48.121243 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.121264 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.121316 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:56:48.122209 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.122234 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.122797 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.123261 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:48.123756 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.124936 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.126221 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.126822 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.126982 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.128056 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:48.128063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:48.128088 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:48.128629 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:48.128712 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:48.128738 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:48.128832 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.129161 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.129596 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":157,"threads_started":1}
I20260927 00:56:48.133966 20163 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22977ca4726427d9d1cae6cc653de00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38407 } }
I20260927 00:56:48.134232 20163 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:48.134267 20163 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22977ca4726427d9d1cae6cc653de00, State: Initialized, Role: FOLLOWER
I20260927 00:56:48.134711 20163 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22977ca4726427d9d1cae6cc653de00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38407 } }
I20260927 00:56:48.134845 20163 raft_consensus.cc:388] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:48.134886 20163 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:48.134963 20163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:48.135965 20163 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22977ca4726427d9d1cae6cc653de00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38407 } }
I20260927 00:56:48.136335 20163 leader_election.cc:302] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a22977ca4726427d9d1cae6cc653de00; no voters:
I20260927 00:56:48.136657 20163 leader_election.cc:288] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:48.136938 20168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:48.137323 20168 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [term 1 LEADER]: Becoming Leader. State: Replica: a22977ca4726427d9d1cae6cc653de00, State: Running, Role: LEADER
I20260927 00:56:48.137784 20168 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22977ca4726427d9d1cae6cc653de00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38407 } }
I20260927 00:56:48.138123 20163 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:48.140003 20169 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a22977ca4726427d9d1cae6cc653de00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22977ca4726427d9d1cae6cc653de00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38407 } } }
I20260927 00:56:48.140020 20170 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22977ca4726427d9d1cae6cc653de00. Latest consensus state: current_term: 1 leader_uuid: "a22977ca4726427d9d1cae6cc653de00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22977ca4726427d9d1cae6cc653de00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38407 } } }
I20260927 00:56:48.140218 20170 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:48.140218 20169 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00 [sys.catalog]: This master's current role is: LEADER
W20260927 00:56:48.143558 20182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:48.143642 20182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:48.143723 20183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:48.144630 20183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:48.151521 20183 catalog_manager.cc:1347] Generated new cluster ID: 9c4ad4295af84e0aa30aea1c50982189
I20260927 00:56:48.151633 20183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:48.174117 20183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:48.175251 20183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:48.180199 20183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: Generated new TSK 0
I20260927 00:56:48.181377 20183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:48.197086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38407
--builtin_ntp_servers=127.0.98.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:56:48.318692 20187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:48.319001 20187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:48.319067 20187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:48.323365 20187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:48.323433 20187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:48.323514 20187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:56:48.328608 20187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:48.329804 20187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:48.331278 20187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:48.342161 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:48.342151 20193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:48.342242 20192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.342473 20187 server_base.cc:1034] running on GCE node
I20260927 00:56:48.343104 20187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:48.343874 20187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:48.345095 20187 hybrid_clock.cc:648] HybridClock initialized: now 1790470608345066 us; error 50 us; skew 500 ppm
I20260927 00:56:48.347473 20187 webserver.cc:460] Webserver started at http://127.0.98.1:45625/ using document root <none> and password file <none>
I20260927 00:56:48.348126 20187 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:48.348198 20187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:48.348408 20187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:48.351637 20187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data/instance:
uuid: "89e1049fda064be5a645aeba7d72004e"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "1d7a6152d0ada2a7780d470def16b15e"
server_key_iv: "98871ae41d87a8834610c28fb0b4641f"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.352411 20187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal/instance:
uuid: "89e1049fda064be5a645aeba7d72004e"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "1d7a6152d0ada2a7780d470def16b15e"
server_key_iv: "98871ae41d87a8834610c28fb0b4641f"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.357854 20187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:56:48.361673 20201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.363792 20187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:56:48.363989 20187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal
uuid: "89e1049fda064be5a645aeba7d72004e"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "1d7a6152d0ada2a7780d470def16b15e"
server_key_iv: "98871ae41d87a8834610c28fb0b4641f"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.364147 20187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:48.382043 20187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:48.383018 20187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:48.383262 20187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:48.384024 20187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:48.385195 20187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:48.385246 20187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.385306 20187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:48.385335 20187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.396680 20187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37545
I20260927 00:56:48.396724 20314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37545 every 8 connection(s)
I20260927 00:56:48.397889 20187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:56:48.404394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20187
I20260927 00:56:48.404569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:56:48.404950 392 external_mini_cluster.cc:1428] Setting key 37504b78fa87888d52276d27c53c9b74
I20260927 00:56:48.407344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38407
--builtin_ntp_servers=127.0.98.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:56:48.409667 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.400162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34111 (local address 127.0.98.62:38407)
0927 00:56:48.400366 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.400371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.401617 (+ 1246us) server_negotiation.cc:407] Connection header received
0927 00:56:48.403333 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.403337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.403419 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.403502 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:48.404907 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.405550 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.407129 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.407426 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.407549 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.409143 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:48.409146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:48.409148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:48.409163 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:48.409184 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:48.409192 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:48.409256 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.409416 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.409492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:56:48.410737 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.400520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38407 (local address 127.0.98.1:34111)
0927 00:56:48.401408 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.401462 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.403041 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.403724 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.403733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.403840 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:48.404667 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.404691 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.405736 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.405739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.406979 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.406993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.407723 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.407730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.407965 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.408781 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:48.408783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:48.408838 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:48.408927 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:48.409274 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:48.409284 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:48.409290 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.409411 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.409866 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":148,"threads_started":1}
I20260927 00:56:48.412290 20315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38407
I20260927 00:56:48.412544 20315 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:48.413308 20315 heartbeater.cc:511] Master 127.0.98.62:38407 requested a full tablet report, sending...
I20260927 00:56:48.415084 20127 ts_manager.cc:194] Registered new tserver with Master: 89e1049fda064be5a645aeba7d72004e (127.0.98.1:37545)
I20260927 00:56:48.416327 20127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34111
W20260927 00:56:48.541111 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:48.541482 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:48.541543 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:48.546066 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:48.546157 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:48.546250 20318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:56:48.551782 20318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:48.553304 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:48.555097 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:48.566953 20323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.567113 20318 server_base.cc:1034] running on GCE node
W20260927 00:56:48.566953 20326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:48.566962 20324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.567925 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:48.568792 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:48.570012 20318 hybrid_clock.cc:648] HybridClock initialized: now 1790470608569980 us; error 45 us; skew 500 ppm
I20260927 00:56:48.572265 20318 webserver.cc:460] Webserver started at http://127.0.98.2:33195/ using document root <none> and password file <none>
I20260927 00:56:48.572887 20318 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:48.572958 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:48.573172 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:48.575476 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data/instance:
uuid: "7e5aa9a66a694c95aae2500c4135ef0d"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "beb654bee61352ad00e26b4598ae1779"
server_key_iv: "9bfe329dce5ce63560d240e6a21c79f6"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.576035 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7e5aa9a66a694c95aae2500c4135ef0d"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "beb654bee61352ad00e26b4598ae1779"
server_key_iv: "9bfe329dce5ce63560d240e6a21c79f6"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.580462 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:56:48.583825 20332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.585554 20318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:56:48.585740 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal
uuid: "7e5aa9a66a694c95aae2500c4135ef0d"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "beb654bee61352ad00e26b4598ae1779"
server_key_iv: "9bfe329dce5ce63560d240e6a21c79f6"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.585980 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:48.605693 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:48.606602 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:48.606810 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:48.607573 20318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:48.608733 20318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:48.608786 20318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.608855 20318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:48.608880 20318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.618961 20318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44835
I20260927 00:56:48.619084 20445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44835 every 8 connection(s)
I20260927 00:56:48.620048 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:56:48.626174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20318
I20260927 00:56:48.626299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:56:48.626573 392 external_mini_cluster.cc:1428] Setting key 949c7e94cc3978872ac8416fb2843d53
I20260927 00:56:48.629065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38407
--builtin_ntp_servers=127.0.98.20:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:56:48.630970 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.622325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34295 (local address 127.0.98.62:38407)
0927 00:56:48.622533 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.622538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.623764 (+ 1226us) server_negotiation.cc:407] Connection header received
0927 00:56:48.625456 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.625459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.625517 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.625613 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:48.626831 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.627456 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.628723 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.628936 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.629022 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.630383 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:48.630388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:48.630391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:48.630409 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:48.630447 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:48.630460 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:48.630547 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.630743 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.630815 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:56:48.632568 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.622727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38407 (local address 127.0.98.2:34295)
0927 00:56:48.623576 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.623628 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.625196 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.625870 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.625880 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.625990 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:48.626675 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.626692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.627621 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.627626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.628589 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.628603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.629083 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.629087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.629308 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.630048 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:48.630050 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:48.630104 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:48.630197 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:48.630551 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:48.630557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:48.630563 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.631280 (+ 717us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.631627 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260927 00:56:48.634434 20446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38407
I20260927 00:56:48.634785 20446 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:48.635423 20446 heartbeater.cc:511] Master 127.0.98.62:38407 requested a full tablet report, sending...
I20260927 00:56:48.636907 20127 ts_manager.cc:194] Registered new tserver with Master: 7e5aa9a66a694c95aae2500c4135ef0d (127.0.98.2:44835)
I20260927 00:56:48.637900 20127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34295
W20260927 00:56:48.763410 20449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:48.763759 20449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:48.763803 20449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:48.768348 20449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:48.768450 20449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:48.768517 20449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:56:48.774081 20449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:48.775600 20449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:48.777326 20449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:48.788902 20457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.789052 20449 server_base.cc:1034] running on GCE node
W20260927 00:56:48.788857 20455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:48.788888 20454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:48.789760 20449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:48.790649 20449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:48.791895 20449 hybrid_clock.cc:648] HybridClock initialized: now 1790470608791869 us; error 46 us; skew 500 ppm
I20260927 00:56:48.794298 20449 webserver.cc:460] Webserver started at http://127.0.98.3:38289/ using document root <none> and password file <none>
I20260927 00:56:48.795018 20449 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:48.795095 20449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:48.795365 20449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:48.798022 20449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data/instance:
uuid: "d553f26095664209a943c092a19f7281"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "ee327b75caed31cbc04459919fc80342"
server_key_iv: "0e19376a91730807420d666433d0de5b"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.798689 20449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d553f26095664209a943c092a19f7281"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "ee327b75caed31cbc04459919fc80342"
server_key_iv: "0e19376a91730807420d666433d0de5b"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.803655 20449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 00:56:48.807267 20463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.809096 20449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:56:48.809273 20449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal
uuid: "d553f26095664209a943c092a19f7281"
format_stamp: "Formatted at 2026-09-27 00:56:48 on dist-test-slave-9fh2"
server_key: "ee327b75caed31cbc04459919fc80342"
server_key_iv: "0e19376a91730807420d666433d0de5b"
server_key_version: "encryptionkey@0"
I20260927 00:56:48.809425 20449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:48.825465 20449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:48.826436 20449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:48.826651 20449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:48.827451 20449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:56:48.828701 20449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:56:48.828755 20449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.828827 20449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:56:48.828855 20449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:48.840711 20449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35233
I20260927 00:56:48.840791 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35233 every 8 connection(s)
I20260927 00:56:48.841887 20449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:56:48.847844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20449
I20260927 00:56:48.847956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:56:48.848196 392 external_mini_cluster.cc:1428] Setting key c418515fe0c71be1ea6e73bbb5e22968
I20260927 00:56:48.852916 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.844001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56993 (local address 127.0.98.62:38407)
0927 00:56:48.844185 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.844189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.845540 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 00:56:48.847138 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.847185 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.847256 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.847354 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:48.848645 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.849236 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.850766 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.850998 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.851093 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.852421 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:48.852424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:48.852426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:48.852440 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:48.852461 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:48.852469 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:48.852588 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.852744 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.852797 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:56:48.854182 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.844380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38407 (local address 127.0.98.3:56993)
0927 00:56:48.845344 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.845400 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.846886 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.847603 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.847612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.847718 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:56:48.848470 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.848489 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.849381 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.849385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.850633 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.850646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.851196 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.851202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.851390 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.852117 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:56:48.852118 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:56:48.852169 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:56:48.852253 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:56:48.852578 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:56:48.852586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:56:48.852593 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.852730 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.853167 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":157,"threads_started":1}
I20260927 00:56:48.855644 20577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38407
I20260927 00:56:48.855968 20577 heartbeater.cc:464] Registering TS with master...
I20260927 00:56:48.856611 20577 heartbeater.cc:511] Master 127.0.98.62:38407 requested a full tablet report, sending...
I20260927 00:56:48.857847 20127 ts_manager.cc:194] Registered new tserver with Master: d553f26095664209a943c092a19f7281 (127.0.98.3:35233)
I20260927 00:56:48.858462 20127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56993
I20260927 00:56:48.863103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:56:48.870270 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.865517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.0.98.62:38407)
0927 00:56:48.865709 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.865714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.865968 (+ 254us) server_negotiation.cc:407] Connection header received
0927 00:56:48.866135 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.866138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.866188 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.866269 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:48.866684 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.867267 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.868240 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.868523 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.868649 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.869805 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:48.869809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:48.869811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:48.869824 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:48.869845 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:48.869853 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:48.869917 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.870052 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.870109 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 00:56:48.873653 20127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:56:48.877079 20127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:56:48.894965 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.888408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35233 (local address 127.0.0.1:40486)
0927 00:56:48.889017 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.889029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.889105 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.889845 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.889848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.889860 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:48.889963 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.889970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.892103 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.892107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.893059 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.893068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.894170 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.894173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.894299 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.894368 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.894507 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.894696 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":62,"thread_start_us":97,"threads_started":1}
I20260927 00:56:48.895210 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.887793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.0.98.1:37545)
0927 00:56:48.888189 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.888194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.888950 (+ 756us) server_negotiation.cc:407] Connection header received
0927 00:56:48.889130 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.889135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.889354 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.889517 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:48.889955 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.891333 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.893346 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.894145 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.894251 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.894698 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.894759 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.894872 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":98,"threads_started":1}
I20260927 00:56:48.895777 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.888607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.0.98.3:35233)
0927 00:56:48.889336 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.889348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.889371 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:56:48.889448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.889454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.889681 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.889869 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:48.890756 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.891973 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.893196 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.894018 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.894145 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.894373 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.895432 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.895585 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":114,"threads_started":1}
I20260927 00:56:48.895843 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.887493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37545 (local address 127.0.0.1:58114)
0927 00:56:48.888752 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.888768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.888978 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.889578 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.889585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.889609 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:48.889797 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.889805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.891473 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.891477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.893191 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.893202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.894279 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.894283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.894384 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.894403 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.894492 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.895721 (+ 1229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":1438,"thread_start_us":128,"threads_started":1}
I20260927 00:56:48.896219 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.887730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.0.98.2:44835)
0927 00:56:48.888328 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.888335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.888781 (+ 446us) server_negotiation.cc:407] Connection header received
0927 00:56:48.889142 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.889147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.889354 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.889517 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:48.890065 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.891332 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.894468 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.895301 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.895412 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.895592 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.895959 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.896062 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":138,"threads_started":1}
I20260927 00:56:48.896641 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.887493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44835 (local address 127.0.0.1:33066)
0927 00:56:48.888628 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.888686 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.888899 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.889693 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.889697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.889715 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:48.889900 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.889908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.891473 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.891477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.893196 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.893202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.895663 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.895667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.895791 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.895816 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.896416 (+ 600us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.896514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":186,"thread_start_us":129,"threads_started":1}
I20260927 00:56:48.898849 20511 tablet_service.cc:1467] Processing CreateTablet for tablet a64e881016834ac98f3e929b77997757 (DEFAULT_TABLE table=test-table [id=88996dbe20b14c6f9dfd883e87990c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:48.898989 20249 tablet_service.cc:1467] Processing CreateTablet for tablet a64e881016834ac98f3e929b77997757 (DEFAULT_TABLE table=test-table [id=88996dbe20b14c6f9dfd883e87990c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:48.900053 20511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64e881016834ac98f3e929b77997757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:48.900256 20249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64e881016834ac98f3e929b77997757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:48.906620 20380 tablet_service.cc:1467] Processing CreateTablet for tablet a64e881016834ac98f3e929b77997757 (DEFAULT_TABLE table=test-table [id=88996dbe20b14c6f9dfd883e87990c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:56:48.908033 20380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64e881016834ac98f3e929b77997757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:48.909612 20594 tablet_bootstrap.cc:492] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Bootstrap starting.
I20260927 00:56:48.912132 20595 tablet_bootstrap.cc:492] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e: Bootstrap starting.
I20260927 00:56:48.912396 20594 tablet_bootstrap.cc:654] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:48.913306 20594 log.cc:824] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:48.914973 20595 tablet_bootstrap.cc:654] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:48.915442 20594 tablet_bootstrap.cc:492] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: No bootstrap required, opened a new log
I20260927 00:56:48.915652 20594 ts_tablet_manager.cc:1397] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:56:48.915977 20595 log.cc:824] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:48.916273 20597 tablet_bootstrap.cc:492] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Bootstrap starting.
I20260927 00:56:48.918056 20595 tablet_bootstrap.cc:492] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e: No bootstrap required, opened a new log
I20260927 00:56:48.918258 20595 ts_tablet_manager.cc:1397] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:56:48.918787 20597 tablet_bootstrap.cc:654] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:48.919739 20597 log.cc:824] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:48.920164 20594 raft_consensus.cc:348] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.920426 20594 raft_consensus.cc:374] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:48.920482 20594 raft_consensus.cc:729] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d553f26095664209a943c092a19f7281, State: Initialized, Role: FOLLOWER
I20260927 00:56:48.920936 20594 consensus_queue.cc:260] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.921967 20594 ts_tablet_manager.cc:1428] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:56:48.922246 20597 tablet_bootstrap.cc:492] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: No bootstrap required, opened a new log
I20260927 00:56:48.922318 20577 heartbeater.cc:503] Master 127.0.98.62:38407 was elected leader, sending a full tablet report...
I20260927 00:56:48.922482 20597 ts_tablet_manager.cc:1397] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:56:48.922462 20595 raft_consensus.cc:348] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.922780 20595 raft_consensus.cc:374] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:48.922988 20595 raft_consensus.cc:729] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e1049fda064be5a645aeba7d72004e, State: Initialized, Role: FOLLOWER
I20260927 00:56:48.923578 20595 consensus_queue.cc:260] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.924368 20315 heartbeater.cc:503] Master 127.0.98.62:38407 was elected leader, sending a full tablet report...
I20260927 00:56:48.924762 20595 ts_tablet_manager.cc:1428] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 00:56:48.926859 20597 raft_consensus.cc:348] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.927191 20597 raft_consensus.cc:374] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:48.927255 20597 raft_consensus.cc:729] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5aa9a66a694c95aae2500c4135ef0d, State: Initialized, Role: FOLLOWER
I20260927 00:56:48.927732 20597 consensus_queue.cc:260] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.928427 20446 heartbeater.cc:503] Master 127.0.98.62:38407 was elected leader, sending a full tablet report...
I20260927 00:56:48.928709 20597 ts_tablet_manager.cc:1428] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:56:48.992250 20601 raft_consensus.cc:482] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:56:48.992466 20601 raft_consensus.cc:504] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:48.993693 20601 leader_election.cc:288] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5aa9a66a694c95aae2500c4135ef0d (127.0.98.2:44835), d553f26095664209a943c092a19f7281 (127.0.98.3:35233)
I20260927 00:56:48.998662 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.994669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46251 (local address 127.0.98.3:35233)
0927 00:56:48.994802 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.994806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.995012 (+ 206us) server_negotiation.cc:407] Connection header received
0927 00:56:48.995114 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.995117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.995193 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.995284 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:48.995637 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.996141 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.997585 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.998032 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.998129 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.998199 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.998427 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.998506 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:56:48.998726 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.994035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44835 (local address 127.0.98.1:35245)
0927 00:56:48.994406 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.994421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.994685 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.995115 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.995117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.995169 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:48.995371 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.995380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.996220 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.996223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.997464 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.997475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.998203 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.998207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.998329 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.998354 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.998437 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.998525 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":144,"thread_start_us":146,"threads_started":1}
I20260927 00:56:48.998696 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.994188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35245 (local address 127.0.98.2:44835)
0927 00:56:48.994398 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:48.994403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:48.994542 (+ 139us) server_negotiation.cc:407] Connection header received
0927 00:56:48.994916 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:48.994921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:48.994990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:48.995076 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:56:48.995508 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.996102 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.997626 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.998056 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.998162 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.998238 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:48.998473 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 00:56:48.998559 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:56:48.998950 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:48.994517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35233 (local address 127.0.98.1:46251)
0927 00:56:48.994878 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:48.994893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:56:48.995012 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:56:48.995311 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:56:48.995313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:56:48.995330 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:56:48.995514 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.995522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.996257 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.996260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.997441 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:56:48.997451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:56:48.998166 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:56:48.998169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:56:48.998278 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:48.998322 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:56:48.998385 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:56:48.998476 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":111,"thread_start_us":103,"threads_started":1}
I20260927 00:56:48.999610 20531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64e881016834ac98f3e929b77997757" candidate_uuid: "89e1049fda064be5a645aeba7d72004e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d553f26095664209a943c092a19f7281" is_pre_election: true
I20260927 00:56:48.999698 20400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64e881016834ac98f3e929b77997757" candidate_uuid: "89e1049fda064be5a645aeba7d72004e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" is_pre_election: true
I20260927 00:56:48.999946 20531 raft_consensus.cc:2454] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e1049fda064be5a645aeba7d72004e in term 0.
I20260927 00:56:49.000087 20400 raft_consensus.cc:2454] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e1049fda064be5a645aeba7d72004e in term 0.
I20260927 00:56:49.000516 20202 leader_election.cc:302] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e1049fda064be5a645aeba7d72004e, d553f26095664209a943c092a19f7281; no voters:
I20260927 00:56:49.000811 20601 raft_consensus.cc:2781] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:56:49.000907 20601 raft_consensus.cc:482] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:56:49.000952 20601 raft_consensus.cc:3037] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:49.002023 20601 raft_consensus.cc:504] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:49.002377 20601 leader_election.cc:288] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [CANDIDATE]: Term 1 election: Requested vote from peers 7e5aa9a66a694c95aae2500c4135ef0d (127.0.98.2:44835), d553f26095664209a943c092a19f7281 (127.0.98.3:35233)
I20260927 00:56:49.002889 20531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64e881016834ac98f3e929b77997757" candidate_uuid: "89e1049fda064be5a645aeba7d72004e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d553f26095664209a943c092a19f7281"
I20260927 00:56:49.002892 20400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64e881016834ac98f3e929b77997757" candidate_uuid: "89e1049fda064be5a645aeba7d72004e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5aa9a66a694c95aae2500c4135ef0d"
I20260927 00:56:49.003046 20531 raft_consensus.cc:3037] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:49.003041 20400 raft_consensus.cc:3037] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:49.004474 20400 raft_consensus.cc:2454] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e1049fda064be5a645aeba7d72004e in term 1.
I20260927 00:56:49.004474 20531 raft_consensus.cc:2454] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e1049fda064be5a645aeba7d72004e in term 1.
I20260927 00:56:49.004889 20202 leader_election.cc:302] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e5aa9a66a694c95aae2500c4135ef0d, 89e1049fda064be5a645aeba7d72004e; no voters:
I20260927 00:56:49.005127 20601 raft_consensus.cc:2781] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:49.005362 20601 raft_consensus.cc:686] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [term 1 LEADER]: Becoming Leader. State: Replica: 89e1049fda064be5a645aeba7d72004e, State: Running, Role: LEADER
I20260927 00:56:49.005776 20601 consensus_queue.cc:237] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } }
I20260927 00:56:49.009835 20127 catalog_manager.cc:5495] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e1049fda064be5a645aeba7d72004e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "89e1049fda064be5a645aeba7d72004e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e1049fda064be5a645aeba7d72004e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20260927 00:56:49.023798 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:49.019883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.0.98.1:37545)
0927 00:56:49.020020 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:49.020024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:49.020039 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:56:49.020159 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:49.020161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:49.020224 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:49.020325 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:56:49.020621 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:49.021138 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:49.022155 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:49.022385 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:49.022470 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:49.022816 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:56:49.023367 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:56:49.023476 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:49.023576 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 00:56:49.023640 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:56:49.031028 20400 raft_consensus.cc:1261] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 1 FOLLOWER]: Refusing update from remote peer 89e1049fda064be5a645aeba7d72004e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:49.031028 20531 raft_consensus.cc:1261] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 1 FOLLOWER]: Refusing update from remote peer 89e1049fda064be5a645aeba7d72004e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:56:49.031899 20601 consensus_queue.cc:1035] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d553f26095664209a943c092a19f7281" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:49.032189 20605 consensus_queue.cc:1035] T a64e881016834ac98f3e929b77997757 P 89e1049fda064be5a645aeba7d72004e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5aa9a66a694c95aae2500c4135ef0d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:56:49.041839 20609 mvcc.cc:205] Tried to move back new op lower bound from 7333767614582267904 to 7333767614490722304. Current Snapshot: MvccSnapshot[applied={T|T < 7333767614582267904}]
I20260927 00:56:49.042207 20610 mvcc.cc:205] Tried to move back new op lower bound from 7333767614582267904 to 7333767614490722304. Current Snapshot: MvccSnapshot[applied={T|T < 7333767614582267904}]
I20260927 00:56:49.052774 20127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:49.053001 20127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:56:49.055922 20127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a22977ca4726427d9d1cae6cc653de00: Sending DeleteTablet for 3 replicas of tablet a64e881016834ac98f3e929b77997757
I20260927 00:56:49.057073 20249 tablet_service.cc:1514] Processing DeleteTablet for tablet a64e881016834ac98f3e929b77997757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:49 UTC) from {username='slave'} at 127.0.0.1:58114
I20260927 00:56:49.057117 20380 tablet_service.cc:1514] Processing DeleteTablet for tablet a64e881016834ac98f3e929b77997757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:49 UTC) from {username='slave'} at 127.0.0.1:33066
I20260927 00:56:49.057672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20187
I20260927 00:56:49.058109 20511 tablet_service.cc:1514] Processing DeleteTablet for tablet a64e881016834ac98f3e929b77997757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:56:49 UTC) from {username='slave'} at 127.0.0.1:40486
I20260927 00:56:49.058676 20620 tablet_replica.cc:330] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: stopping tablet replica
I20260927 00:56:49.058966 20619 tablet_replica.cc:330] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: stopping tablet replica
I20260927 00:56:49.059072 20620 raft_consensus.cc:2229] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:49.059427 20619 raft_consensus.cc:2229] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:56:49.059480 20620 raft_consensus.cc:2258] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:49.059844 20619 raft_consensus.cc:2258] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:56:49.061772 20619 ts_tablet_manager.cc:1905] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:49.061772 20620 ts_tablet_manager.cc:1905] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:56:49.064505 20619 ts_tablet_manager.cc:1918] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:49.064505 20620 ts_tablet_manager.cc:1918] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:56:49.064611 20620 log.cc:1196] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-2/wal/wals/a64e881016834ac98f3e929b77997757
I20260927 00:56:49.064711 20619 log.cc:1196] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Deleting WAL directory at /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470348013064-392-0/minicluster-data/ts-1/wal/wals/a64e881016834ac98f3e929b77997757
I20260927 00:56:49.064925 20620 ts_tablet_manager.cc:1939] T a64e881016834ac98f3e929b77997757 P d553f26095664209a943c092a19f7281: Deleting consensus metadata
I20260927 00:56:49.064975 20619 ts_tablet_manager.cc:1939] T a64e881016834ac98f3e929b77997757 P 7e5aa9a66a694c95aae2500c4135ef0d: Deleting consensus metadata
W20260927 00:56:49.065793 20113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37545 (error 108)
I20260927 00:56:49.066007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20318
I20260927 00:56:49.067111 20111 catalog_manager.cc:4841] TS d553f26095664209a943c092a19f7281 (127.0.98.3:35233): tablet a64e881016834ac98f3e929b77997757 (table test-table [id=88996dbe20b14c6f9dfd883e87990c8a]) successfully deleted
I20260927 00:56:49.068662 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:49.068240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37545 (local address 127.0.0.1:58120)
0927 00:56:49.068434 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:49.068541 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 00:56:49.068846 20113 catalog_manager.cc:4584] TS 89e1049fda064be5a645aeba7d72004e (127.0.98.1:37545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a64e881016834ac98f3e929b77997757: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37545: connect: Connection refused (error 111)
I20260927 00:56:49.072638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20449
I20260927 00:56:49.074486 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:49.072798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44835 (local address 127.0.0.1:33070)
0927 00:56:49.074337 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0927 00:56:49.074388 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1460}
I20260927 00:56:49.078832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20096
2026-09-27T00:56:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1232 ms)
[----------] 6 tests from EncryptionPolicyTest (10954 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:49 dist-test-slave-9fh2 krb5kdc[20624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:49 dist-test-slave-9fh2 krb5kdc[20624](info): set up 2 sockets
Sep 27 00:56:49 dist-test-slave-9fh2 krb5kdc[20624](info): commencing operation
krb5kdc: starting...
W20260927 00:56:50.055751 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:50 dist-test-slave-9fh2 krb5kdc[20624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:50.186148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:50.324538 20637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:50.324946 20637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:50.325013 20637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:50.329707 20637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:50.329903 20637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:50.329950 20637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:50.329984 20637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:50.330015 20637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:50.336541 20637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:50.338359 20637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:50.339954 20637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:56:50.350512 20637 server_base.cc:1034] running on GCE node
W20260927 00:56:50.350445 20642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:50.350445 20643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:50.350485 20645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:50.351771 20637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:50.353353 20637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:50.354573 20637 hybrid_clock.cc:648] HybridClock initialized: now 1790470610354543 us; error 52 us; skew 500 ppm
Sep 27 00:56:50 dist-test-slave-9fh2 krb5kdc[20624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:50.358534 20637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:50.360075 20637 webserver.cc:460] Webserver started at http://127.0.0.1:36993/ using document root <none> and password file <none>
I20260927 00:56:50.360858 20637 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:50.360940 20637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:50.361207 20637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:50.364017 20637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "4b8e12979bdd4643bf44737834ef8702"
format_stamp: "Formatted at 2026-09-27 00:56:50 on dist-test-slave-9fh2"
server_key: "e7d77b8a9ede71d58f705c2b72d240b2"
server_key_iv: "74925608aa0439f3f8429f4252906f74"
server_key_version: "encryptionkey@0"
I20260927 00:56:50.364729 20637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b8e12979bdd4643bf44737834ef8702"
format_stamp: "Formatted at 2026-09-27 00:56:50 on dist-test-slave-9fh2"
server_key: "e7d77b8a9ede71d58f705c2b72d240b2"
server_key_iv: "74925608aa0439f3f8429f4252906f74"
server_key_version: "encryptionkey@0"
I20260927 00:56:50.370103 20637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:56:50.373888 20652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:50.375973 20637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:56:50.376175 20637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "4b8e12979bdd4643bf44737834ef8702"
format_stamp: "Formatted at 2026-09-27 00:56:50 on dist-test-slave-9fh2"
server_key: "e7d77b8a9ede71d58f705c2b72d240b2"
server_key_iv: "74925608aa0439f3f8429f4252906f74"
server_key_version: "encryptionkey@0"
I20260927 00:56:50.376363 20637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:50.402087 20637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:50.406067 20637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:50.406296 20637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:50.414938 20637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34027
I20260927 00:56:50.415055 20704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34027 every 8 connection(s)
I20260927 00:56:50.416186 20637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:50.419324 20705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:50.424925 20705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702: Bootstrap starting.
I20260927 00:56:50.425354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20637
I20260927 00:56:50.425486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:50.425786 392 external_mini_cluster.cc:1428] Setting key cdfd51a0b4f45bffa55a760158f86a98
I20260927 00:56:50.427896 20705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:50.428644 20705 log.cc:824] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:50.430581 20705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702: No bootstrap required, opened a new log
I20260927 00:56:50.434288 20705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8e12979bdd4643bf44737834ef8702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260927 00:56:50.434628 20705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:50.434688 20705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8e12979bdd4643bf44737834ef8702, State: Initialized, Role: FOLLOWER
I20260927 00:56:50.435177 20705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8e12979bdd4643bf44737834ef8702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260927 00:56:50.435316 20705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:50.435379 20705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:50.435478 20705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:56:50 dist-test-slave-9fh2 krb5kdc[20624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:56:50.436689 20705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8e12979bdd4643bf44737834ef8702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260927 00:56:50.437088 20705 leader_election.cc:302] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b8e12979bdd4643bf44737834ef8702; no voters:
I20260927 00:56:50.437461 20705 leader_election.cc:288] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:50.437651 20710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:50.438014 20710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [term 1 LEADER]: Becoming Leader. State: Replica: 4b8e12979bdd4643bf44737834ef8702, State: Running, Role: LEADER
I20260927 00:56:50.438510 20710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8e12979bdd4643bf44737834ef8702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260927 00:56:50.438925 20705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:50.440785 20711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b8e12979bdd4643bf44737834ef8702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8e12979bdd4643bf44737834ef8702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } } }
I20260927 00:56:50.440977 20711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:50.441457 20712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b8e12979bdd4643bf44737834ef8702. Latest consensus state: current_term: 1 leader_uuid: "4b8e12979bdd4643bf44737834ef8702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8e12979bdd4643bf44737834ef8702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } } }
I20260927 00:56:50.441574 20712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:50.444047 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:50.427373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34633 (local address 127.0.0.1:34027)
0927 00:56:50.427921 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:50.427945 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:50.428004 (+ 59us) server_negotiation.cc:407] Connection header received
0927 00:56:50.428983 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:50.429009 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:50.429506 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:50.429932 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:50.430935 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:50.432115 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:50.433059 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:50.433634 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:50.433805 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:50.437413 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:50.437422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:50.437459 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:50.437520 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:50.439874 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:50.440705 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:50.440713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:50.440722 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:50.440895 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:50.441314 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:50.441318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:50.441320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:50.442010 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:50.442267 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:50.442704 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 00:56:50.443012 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":166,"threads_started":1}
W20260927 00:56:50.444561 20724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:50.444662 20724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:50.444730 20725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:50.445559 20725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:50.448093 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:56:50.452694 20725 catalog_manager.cc:1347] Generated new cluster ID: 3f86d0b8c5804fc5bb01ba43120ce378
I20260927 00:56:50.452788 20725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:50.460665 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:50.450809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53381 (local address 127.0.0.1:34027)
0927 00:56:50.450930 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:50.450935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:50.451351 (+ 416us) server_negotiation.cc:407] Connection header received
0927 00:56:50.451503 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:50.451507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:50.451567 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:50.451672 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:50.452553 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:50.453273 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:50.455426 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:50.455686 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:50.455781 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:50.457364 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:50.457368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:50.457370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:50.457398 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:50.458794 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:50.459310 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:50.459314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:50.459316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:50.459396 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:50.459713 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:50.459716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:50.459717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:50.459919 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:50.460068 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:50.460337 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 00:56:50.460444 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:56:50.473515 20725 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:50.474761 20725 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:50.482332 20725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b8e12979bdd4643bf44737834ef8702: Generated new TSK 0
I20260927 00:56:50.483362 20725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:50.494958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20637
W20260927 00:56:50.501281 20622 connection.cc:535] client connection to 127.0.0.1:34027 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34027): Connection reset by peer (error 104)
2026-09-27T00:56:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:50 dist-test-slave-9fh2 krb5kdc[20736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:50 dist-test-slave-9fh2 krb5kdc[20736](info): set up 2 sockets
Sep 27 00:56:50 dist-test-slave-9fh2 krb5kdc[20736](info): commencing operation
krb5kdc: starting...
W20260927 00:56:51.463503 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:51 dist-test-slave-9fh2 krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:51.597962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:51.730439 20749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:51.730794 20749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:51.730858 20749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:51.735194 20749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:51.735286 20749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:51.735314 20749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:51.735335 20749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:51.735364 20749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:51.741251 20749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:51.742693 20749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:51.743978 20749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:51.754019 20754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:51.754042 20755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:51.754181 20749 server_base.cc:1034] running on GCE node
W20260927 00:56:51.754042 20757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:51.755065 20749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:51.756510 20749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:51.757735 20749 hybrid_clock.cc:648] HybridClock initialized: now 1790470611757716 us; error 38 us; skew 500 ppm
Sep 27 00:56:51 dist-test-slave-9fh2 krb5kdc[20736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:51.768245 20749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:51.769662 20749 webserver.cc:460] Webserver started at http://127.0.0.1:42501/ using document root <none> and password file <none>
I20260927 00:56:51.770402 20749 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:51.770485 20749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:51.770731 20749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:51.773416 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "a46137aaa9a94e59bbdac8961f3942ef"
format_stamp: "Formatted at 2026-09-27 00:56:51 on dist-test-slave-9fh2"
server_key: "eafa0d9d6ea4a9d2cc1cc9d5a780376e"
server_key_iv: "5d3fe21424bc67cae7f337ed0500116b"
server_key_version: "encryptionkey@0"
I20260927 00:56:51.774135 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "a46137aaa9a94e59bbdac8961f3942ef"
format_stamp: "Formatted at 2026-09-27 00:56:51 on dist-test-slave-9fh2"
server_key: "eafa0d9d6ea4a9d2cc1cc9d5a780376e"
server_key_iv: "5d3fe21424bc67cae7f337ed0500116b"
server_key_version: "encryptionkey@0"
I20260927 00:56:51.779207 20749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 00:56:51.782708 20764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:51.784391 20749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:51.784554 20749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "a46137aaa9a94e59bbdac8961f3942ef"
format_stamp: "Formatted at 2026-09-27 00:56:51 on dist-test-slave-9fh2"
server_key: "eafa0d9d6ea4a9d2cc1cc9d5a780376e"
server_key_iv: "5d3fe21424bc67cae7f337ed0500116b"
server_key_version: "encryptionkey@0"
I20260927 00:56:51.784699 20749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:51.800230 20749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:51.804126 20749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:51.804352 20749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:51.812856 20749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39337
I20260927 00:56:51.812896 20816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39337 every 8 connection(s)
I20260927 00:56:51.814029 20749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:51.816179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20749
I20260927 00:56:51.816318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:51.816637 392 external_mini_cluster.cc:1428] Setting key c0d027b7448e83f8e636e3ff8daa1d44
I20260927 00:56:51.817222 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:51.824199 20817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef: Bootstrap starting.
I20260927 00:56:51.826592 20817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:51.827392 20817 log.cc:824] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:51.828364 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:51.818378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59619 (local address 127.0.0.1:39337)
0927 00:56:51.818960 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:51.818991 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:51.819048 (+ 57us) server_negotiation.cc:407] Connection header received
0927 00:56:51.819941 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:51.819964 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:51.820483 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:51.821104 (+ 621us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:51.821648 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:51.822813 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:51.823873 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:51.824487 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:51.824674 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:51.825848 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:51.825856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:51.825873 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:51.826399 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:51.826476 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:51.826513 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:51.826659 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:51.827133 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 00:56:51.827350 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":184,"threads_started":1}
I20260927 00:56:51.829594 20817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef: No bootstrap required, opened a new log
I20260927 00:56:51.832370 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:56:51.833484 20817 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46137aaa9a94e59bbdac8961f3942ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260927 00:56:51.833740 20817 raft_consensus.cc:374] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:51.833809 20817 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46137aaa9a94e59bbdac8961f3942ef, State: Initialized, Role: FOLLOWER
I20260927 00:56:51.834255 20817 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46137aaa9a94e59bbdac8961f3942ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260927 00:56:51.834391 20817 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:51.834458 20817 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:51.834561 20817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:51.835656 20817 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46137aaa9a94e59bbdac8961f3942ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260927 00:56:51.836011 20817 leader_election.cc:302] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46137aaa9a94e59bbdac8961f3942ef; no voters:
I20260927 00:56:51.836300 20817 leader_election.cc:288] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:51.836383 20827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:51.836560 20827 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [term 1 LEADER]: Becoming Leader. State: Replica: a46137aaa9a94e59bbdac8961f3942ef, State: Running, Role: LEADER
I20260927 00:56:51.836866 20827 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46137aaa9a94e59bbdac8961f3942ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260927 00:56:51.837677 20817 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:51.839061 20828 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a46137aaa9a94e59bbdac8961f3942ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46137aaa9a94e59bbdac8961f3942ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260927 00:56:51.839232 20828 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:51.839740 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46137aaa9a94e59bbdac8961f3942ef. Latest consensus state: current_term: 1 leader_uuid: "a46137aaa9a94e59bbdac8961f3942ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46137aaa9a94e59bbdac8961f3942ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260927 00:56:51.839846 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:51.840068 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:51.834844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54271 (local address 127.0.0.1:39337)
0927 00:56:51.834977 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:51.834983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:51.835233 (+ 250us) server_negotiation.cc:407] Connection header received
0927 00:56:51.835404 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:51.835408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:51.835462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:51.835531 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:51.835847 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:51.836557 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:51.837674 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:51.838006 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:51.838593 (+ 587us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:51.839220 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:51.839224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:51.839227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:51.839244 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:51.839276 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:51.839290 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:51.839384 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:51.839860 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 00:56:51.839907 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:56:51.843561 20842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:51.843642 20842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:51.843743 20836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:51.844521 20836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:51.850080 20836 catalog_manager.cc:1347] Generated new cluster ID: 573b2e7816f1431895a2d8260f10a604
I20260927 00:56:51.850170 20836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:51.855211 20836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:51.856760 20836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:51.869866 20836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a46137aaa9a94e59bbdac8961f3942ef: Generated new TSK 0
I20260927 00:56:51.870705 20836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:51.883445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20749
W20260927 00:56:51.888564 20734 connection.cc:535] client connection to 127.0.0.1:39337 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39337): Connection reset by peer (error 104)
2026-09-27T00:56:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:51 dist-test-slave-9fh2 krb5kdc[20848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:51 dist-test-slave-9fh2 krb5kdc[20848](info): set up 2 sockets
Sep 27 00:56:51 dist-test-slave-9fh2 krb5kdc[20848](info): commencing operation
krb5kdc: starting...
W20260927 00:56:52.851784 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:52 dist-test-slave-9fh2 krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:52.988323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:53.104368 20861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:53.104655 20861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:53.104712 20861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:53.108738 20861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:53.108814 20861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:53.108840 20861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:53.108861 20861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:53.108889 20861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:53.114267 20861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:53.115398 20861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:53.116408 20861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:53.125012 20866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:53.125157 20867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:53.125166 20869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:53.125450 20861 server_base.cc:1034] running on GCE node
I20260927 00:56:53.126057 20861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:53.127357 20861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:53.128530 20861 hybrid_clock.cc:648] HybridClock initialized: now 1790470613128499 us; error 60 us; skew 500 ppm
Sep 27 00:56:53 dist-test-slave-9fh2 krb5kdc[20848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:53.131474 20861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:53.132465 20861 webserver.cc:460] Webserver started at http://127.0.0.1:39913/ using document root <none> and password file <none>
I20260927 00:56:53.133041 20861 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:53.133106 20861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:53.133308 20861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:53.135397 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c760432a2cd34b12b9fc9d67cdf6349d"
format_stamp: "Formatted at 2026-09-27 00:56:53 on dist-test-slave-9fh2"
server_key: "7efe93e58643a840e4625a88e7dd8ebe"
server_key_iv: "fa3b38e8217b1bea14fb294125a45396"
server_key_version: "encryptionkey@0"
I20260927 00:56:53.135979 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c760432a2cd34b12b9fc9d67cdf6349d"
format_stamp: "Formatted at 2026-09-27 00:56:53 on dist-test-slave-9fh2"
server_key: "7efe93e58643a840e4625a88e7dd8ebe"
server_key_iv: "fa3b38e8217b1bea14fb294125a45396"
server_key_version: "encryptionkey@0"
I20260927 00:56:53.140105 20861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:56:53.142716 20876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:53.144310 20861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:56:53.144450 20861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c760432a2cd34b12b9fc9d67cdf6349d"
format_stamp: "Formatted at 2026-09-27 00:56:53 on dist-test-slave-9fh2"
server_key: "7efe93e58643a840e4625a88e7dd8ebe"
server_key_iv: "fa3b38e8217b1bea14fb294125a45396"
server_key_version: "encryptionkey@0"
I20260927 00:56:53.144559 20861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:53.149202 20861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:53.152288 20861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:53.152460 20861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:53.160086 20861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35269
I20260927 00:56:53.160147 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35269 every 8 connection(s)
I20260927 00:56:53.161180 20861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:53.164870 20929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:53.166237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20861
I20260927 00:56:53.166390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:53.166702 392 external_mini_cluster.cc:1428] Setting key 54d4b9cfac69826ace4870a2cdf7a494
I20260927 00:56:53.173000 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:53.168521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44371 (local address 127.0.0.1:35269)
0927 00:56:53.169175 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:53.169194 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:53.169240 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:56:53.170080 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:53.170100 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:53.170452 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:53.170868 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:53.171771 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:53.171776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:53.171787 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:53.171798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:53.171855 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:53.171875 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:53.172017 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:53.172085 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:56:53.172245 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":209,"threads_started":1}
I20260927 00:56:53.173256 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d: Bootstrap starting.
I20260927 00:56:53.176410 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:56:53.176431 20929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:53.177261 20929 log.cc:824] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:53.179943 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d: No bootstrap required, opened a new log
I20260927 00:56:53.180096 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:53.178417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35585 (local address 127.0.0.1:35269)
0927 00:56:53.178542 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:53.178546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:53.178657 (+ 111us) server_negotiation.cc:407] Connection header received
0927 00:56:53.178767 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:53.178770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:53.178816 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:53.178919 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:53.179654 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:53.179657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:53.179661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:53.179674 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:53.179708 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:53.179721 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:53.179811 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:53.179892 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 00:56:53.179947 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:56:53.184139 20929 raft_consensus.cc:348] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260927 00:56:53.184427 20929 raft_consensus.cc:374] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:53.184471 20929 raft_consensus.cc:729] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c760432a2cd34b12b9fc9d67cdf6349d, State: Initialized, Role: FOLLOWER
I20260927 00:56:53.184908 20929 consensus_queue.cc:260] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260927 00:56:53.185030 20929 raft_consensus.cc:388] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:53.185081 20929 raft_consensus.cc:482] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:53.185158 20929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:53.186275 20929 raft_consensus.cc:504] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260927 00:56:53.186671 20929 leader_election.cc:302] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c760432a2cd34b12b9fc9d67cdf6349d; no voters:
I20260927 00:56:53.187017 20929 leader_election.cc:288] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:53.187263 20939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:53.187638 20939 raft_consensus.cc:686] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [term 1 LEADER]: Becoming Leader. State: Replica: c760432a2cd34b12b9fc9d67cdf6349d, State: Running, Role: LEADER
I20260927 00:56:53.188017 20939 consensus_queue.cc:237] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } }
I20260927 00:56:53.188607 20929 sys_catalog.cc:564] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:53.190114 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c760432a2cd34b12b9fc9d67cdf6349d. Latest consensus state: current_term: 1 leader_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } } }
I20260927 00:56:53.190053 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c760432a2cd34b12b9fc9d67cdf6349d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35269 } } }
I20260927 00:56:53.190238 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:53.190238 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:53.190853 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:53.194814 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:53.202427 20948 catalog_manager.cc:1347] Generated new cluster ID: be56aea798d04c67a7c984ba5e6aae6b
I20260927 00:56:53.202543 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:53.229964 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:53.230324 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:53.234542 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c760432a2cd34b12b9fc9d67cdf6349d: Generated new TSK 0
I20260927 00:56:53.235330 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:53.248404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20861
2026-09-27T00:56:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:53 dist-test-slave-9fh2 krb5kdc[20960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:53 dist-test-slave-9fh2 krb5kdc[20960](info): set up 2 sockets
Sep 27 00:56:53 dist-test-slave-9fh2 krb5kdc[20960](info): commencing operation
krb5kdc: starting...
W20260927 00:56:54.223537 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:54 dist-test-slave-9fh2 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:54.360327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:54.471040 20973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:54.471369 20973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:54.471412 20973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:54.475417 20973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:54.475490 20973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:54.475509 20973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:54.475526 20973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:54.475541 20973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:54.481056 20973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:54.482216 20973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:54.483217 20973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:54.492913 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:54.492944 20979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:54.492934 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:54.493187 20973 server_base.cc:1034] running on GCE node
I20260927 00:56:54.493813 20973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:54.494900 20973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:54.496079 20973 hybrid_clock.cc:648] HybridClock initialized: now 1790470614496026 us; error 77 us; skew 500 ppm
Sep 27 00:56:54 dist-test-slave-9fh2 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:54.499341 20973 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:54.500532 20973 webserver.cc:460] Webserver started at http://127.0.0.1:41131/ using document root <none> and password file <none>
I20260927 00:56:54.501204 20973 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:54.501269 20973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:54.501492 20973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:54.504016 20973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "88d746ff97b44145aa6cc45460386f7b"
format_stamp: "Formatted at 2026-09-27 00:56:54 on dist-test-slave-9fh2"
server_key: "376c1be889995326796e10ba0e1cce8a"
server_key_iv: "575426f18c80c83d6f6ec8635db48153"
server_key_version: "encryptionkey@0"
I20260927 00:56:54.504640 20973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "88d746ff97b44145aa6cc45460386f7b"
format_stamp: "Formatted at 2026-09-27 00:56:54 on dist-test-slave-9fh2"
server_key: "376c1be889995326796e10ba0e1cce8a"
server_key_iv: "575426f18c80c83d6f6ec8635db48153"
server_key_version: "encryptionkey@0"
I20260927 00:56:54.509199 20973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:56:54.512394 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:54.514390 20973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:56:54.514599 20973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "88d746ff97b44145aa6cc45460386f7b"
format_stamp: "Formatted at 2026-09-27 00:56:54 on dist-test-slave-9fh2"
server_key: "376c1be889995326796e10ba0e1cce8a"
server_key_iv: "575426f18c80c83d6f6ec8635db48153"
server_key_version: "encryptionkey@0"
I20260927 00:56:54.514778 20973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:54.540827 20973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:54.544780 20973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:54.545007 20973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:54.553444 20973 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36897
I20260927 00:56:54.553546 21040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36897 every 8 connection(s)
I20260927 00:56:54.554699 20973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:54.557847 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:54.558338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 20973
I20260927 00:56:54.558476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:54.558802 392 external_mini_cluster.cc:1428] Setting key 1d4631c2a3b3790c53443a902436e4a0
I20260927 00:56:54.564757 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b: Bootstrap starting.
I20260927 00:56:54.567008 21041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:54.567734 21041 log.cc:824] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:56:54 dist-test-slave-9fh2 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:56:54.569617 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b: No bootstrap required, opened a new log
I20260927 00:56:54.573241 21041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d746ff97b44145aa6cc45460386f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36897 } }
I20260927 00:56:54.573489 21041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:54.573554 21041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d746ff97b44145aa6cc45460386f7b, State: Initialized, Role: FOLLOWER
I20260927 00:56:54.574077 21041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d746ff97b44145aa6cc45460386f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36897 } }
I20260927 00:56:54.574255 21041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:54.574323 21041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:54.574419 21041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:54.575243 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:54.560684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48425 (local address 127.0.0.1:36897)
0927 00:56:54.561243 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:54.561265 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:54.561311 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:56:54.562129 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:54.562154 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:54.562589 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:54.562981 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:54.564131 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:54.565365 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:54.566297 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:54.566876 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:54.567062 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:54.569770 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:54.569777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:54.569802 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:54.569839 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:54.571842 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:54.572267 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:54.572272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:54.572278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:54.572400 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:54.572669 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:54.572673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:54.572674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:54.573246 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:54.573472 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:54.573833 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 00:56:54.574199 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":162,"threads_started":1}
I20260927 00:56:54.575621 21041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d746ff97b44145aa6cc45460386f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36897 } }
I20260927 00:56:54.575968 21041 leader_election.cc:302] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d746ff97b44145aa6cc45460386f7b; no voters:
I20260927 00:56:54.576270 21041 leader_election.cc:288] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:54.576450 21046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:54.576882 21046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [term 1 LEADER]: Becoming Leader. State: Replica: 88d746ff97b44145aa6cc45460386f7b, State: Running, Role: LEADER
I20260927 00:56:54.577332 21046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d746ff97b44145aa6cc45460386f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36897 } }
I20260927 00:56:54.577665 21041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:54.579173 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:56:54.579532 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d746ff97b44145aa6cc45460386f7b. Latest consensus state: current_term: 1 leader_uuid: "88d746ff97b44145aa6cc45460386f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d746ff97b44145aa6cc45460386f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36897 } } }
I20260927 00:56:54.579715 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:54.580005 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d746ff97b44145aa6cc45460386f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d746ff97b44145aa6cc45460386f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36897 } } }
I20260927 00:56:54.580225 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b [sys.catalog]: This master's current role is: LEADER
W20260927 00:56:54.583819 21061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:54.583912 21061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:54.583987 21059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:54.584894 21059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:54.591650 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:54.582038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58881 (local address 127.0.0.1:36897)
0927 00:56:54.582182 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:54.582188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:54.582585 (+ 397us) server_negotiation.cc:407] Connection header received
0927 00:56:54.582722 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:54.582726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:54.582785 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:54.582896 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:54.583748 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:54.584518 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:54.585795 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:54.586147 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:54.586260 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:54.588047 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:54.588053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:54.588055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:54.588092 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:54.589770 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:54.590355 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:54.590359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:54.590361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:54.590457 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:54.590704 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:54.590708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:54.590710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:54.590915 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:54.591045 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:54.591314 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 00:56:54.591454 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:56:54.592209 21059 catalog_manager.cc:1347] Generated new cluster ID: 03dd55ea99fd4802af5e8cb82e56dcc4
I20260927 00:56:54.592272 21059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:54.605057 21059 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:54.606292 21059 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:54.616195 21059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88d746ff97b44145aa6cc45460386f7b: Generated new TSK 0
I20260927 00:56:54.617118 21059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:54.626783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 20973
2026-09-27T00:56:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:54 dist-test-slave-9fh2 krb5kdc[21072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:54 dist-test-slave-9fh2 krb5kdc[21072](info): set up 2 sockets
Sep 27 00:56:54 dist-test-slave-9fh2 krb5kdc[21072](info): commencing operation
krb5kdc: starting...
W20260927 00:56:55.595672 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:55 dist-test-slave-9fh2 krb5kdc[21072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:55.730786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:55.839879 21085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:55.840181 21085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:55.840242 21085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:55.844395 21085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:55.844475 21085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:55.844501 21085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:55.844523 21085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:55.844552 21085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:55.849995 21085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:55.851130 21085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:55.852202 21085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:55.861274 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:55.861364 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:55.861263 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:55.861793 21085 server_base.cc:1034] running on GCE node
I20260927 00:56:55.862385 21085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:55.863571 21085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:55.864773 21085 hybrid_clock.cc:648] HybridClock initialized: now 1790470615864747 us; error 48 us; skew 500 ppm
Sep 27 00:56:55 dist-test-slave-9fh2 krb5kdc[21072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:55.867676 21085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:55.868691 21085 webserver.cc:460] Webserver started at http://127.0.0.1:42773/ using document root <none> and password file <none>
I20260927 00:56:55.869275 21085 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:55.869340 21085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:55.869534 21085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:55.871680 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa"
format_stamp: "Formatted at 2026-09-27 00:56:55 on dist-test-slave-9fh2"
server_key: "b4caa7d0253d597bb016545a37f46ca5"
server_key_iv: "9c7390da1965391982ad62748b17ec79"
server_key_version: "encryptionkey@0"
I20260927 00:56:55.872217 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa"
format_stamp: "Formatted at 2026-09-27 00:56:55 on dist-test-slave-9fh2"
server_key: "b4caa7d0253d597bb016545a37f46ca5"
server_key_iv: "9c7390da1965391982ad62748b17ec79"
server_key_version: "encryptionkey@0"
I20260927 00:56:55.876353 21085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:56:55.879354 21100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:55.880957 21085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:56:55.881094 21085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa"
format_stamp: "Formatted at 2026-09-27 00:56:55 on dist-test-slave-9fh2"
server_key: "b4caa7d0253d597bb016545a37f46ca5"
server_key_iv: "9c7390da1965391982ad62748b17ec79"
server_key_version: "encryptionkey@0"
I20260927 00:56:55.881208 21085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:55.911389 21085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:55.914661 21085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:55.914857 21085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:55.922410 21085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40721
I20260927 00:56:55.922477 21152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40721 every 8 connection(s)
I20260927 00:56:55.923533 21085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:55.927467 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:55.929077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21085
I20260927 00:56:55.929298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:55.929701 392 external_mini_cluster.cc:1428] Setting key 9ee08dfa0f1773519a3c7e701dde468f
I20260927 00:56:55.936439 21153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa: Bootstrap starting.
I20260927 00:56:55.939934 21153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:55.940907 21153 log.cc:824] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:55.940989 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:55.931680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58559 (local address 127.0.0.1:40721)
0927 00:56:55.932203 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:55.932220 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:55.932272 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:56:55.933107 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:55.933129 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:55.933535 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:55.933869 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:55.934462 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:55.935621 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:55.936977 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:55.937496 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:55.937693 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:55.938782 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:55.938787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:55.938803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:55.939321 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:55.939389 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:55.939418 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:55.939514 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:55.939693 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 00:56:55.940098 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
I20260927 00:56:55.943722 21153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa: No bootstrap required, opened a new log
I20260927 00:56:55.944720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:56:55.948447 21153 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260927 00:56:55.948764 21153 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:55.948982 21153 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a1e5146cde4457b4bf2c2b5ae50caa, State: Initialized, Role: FOLLOWER
I20260927 00:56:55.949514 21153 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260927 00:56:55.949673 21153 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:55.949749 21153 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:55.949870 21153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:55.951383 21153 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260927 00:56:55.951793 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:55.947006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35335 (local address 127.0.0.1:40721)
0927 00:56:55.947197 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:55.947202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:55.947341 (+ 139us) server_negotiation.cc:407] Connection header received
0927 00:56:55.947560 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:55.947564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:55.947626 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:55.947729 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:55.948128 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:55.948689 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:55.949572 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:55.949929 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:55.950033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:55.951125 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:55.951127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:55.951130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:55.951176 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:55.951212 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:55.951226 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:55.951296 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:55.951506 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 00:56:55.951670 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:56:55.951857 21153 leader_election.cc:302] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a1e5146cde4457b4bf2c2b5ae50caa; no voters:
I20260927 00:56:55.952276 21153 leader_election.cc:288] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:55.952418 21163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:55.952696 21163 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [term 1 LEADER]: Becoming Leader. State: Replica: f5a1e5146cde4457b4bf2c2b5ae50caa, State: Running, Role: LEADER
I20260927 00:56:55.953197 21163 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260927 00:56:55.953830 21153 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:55.954506 21163 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a1e5146cde4457b4bf2c2b5ae50caa. Latest consensus state: current_term: 1 leader_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } } }
I20260927 00:56:55.954658 21163 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:55.955071 21169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:55.955245 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1e5146cde4457b4bf2c2b5ae50caa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } } }
I20260927 00:56:55.955471 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:55.958882 21169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:55.967365 21169 catalog_manager.cc:1347] Generated new cluster ID: f185d152aa304a56aad4e5d71c1ce52a
I20260927 00:56:55.967470 21169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:55.973543 21169 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:55.975101 21169 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:55.982929 21169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5a1e5146cde4457b4bf2c2b5ae50caa: Generated new TSK 0
I20260927 00:56:55.984094 21169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:55.993532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21085
2026-09-27T00:56:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:56 dist-test-slave-9fh2 krb5kdc[21184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:56 dist-test-slave-9fh2 krb5kdc[21184](info): set up 2 sockets
Sep 27 00:56:56 dist-test-slave-9fh2 krb5kdc[21184](info): commencing operation
krb5kdc: starting...
W20260927 00:56:56.971714 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:57 dist-test-slave-9fh2 krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:57.110636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:57.227447 21197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:57.227751 21197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:57.227794 21197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:57.232019 21197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:57.232110 21197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:57.232128 21197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:57.232144 21197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:57.232159 21197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:57.238430 21197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:57.239665 21197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:57.240713 21197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:57.250594 21205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:57.250536 21203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:57.250536 21202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:57.250669 21197 server_base.cc:1034] running on GCE node
I20260927 00:56:57.251708 21197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:57.252934 21197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:57.254129 21197 hybrid_clock.cc:648] HybridClock initialized: now 1790470617254117 us; error 44 us; skew 500 ppm
Sep 27 00:56:57 dist-test-slave-9fh2 krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:57.262414 21197 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:57.264168 21197 webserver.cc:460] Webserver started at http://127.0.0.1:43141/ using document root <none> and password file <none>
I20260927 00:56:57.264890 21197 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:57.264973 21197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:57.265204 21197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:57.267443 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "85d4826d624c4290a478666406b1f2b9"
format_stamp: "Formatted at 2026-09-27 00:56:57 on dist-test-slave-9fh2"
server_key: "2b2bc2e660aa4a98dcb41abad9cc27b1"
server_key_iv: "9f7212ba7a61553c47c828420e98164c"
server_key_version: "encryptionkey@0"
I20260927 00:56:57.268108 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "85d4826d624c4290a478666406b1f2b9"
format_stamp: "Formatted at 2026-09-27 00:56:57 on dist-test-slave-9fh2"
server_key: "2b2bc2e660aa4a98dcb41abad9cc27b1"
server_key_iv: "9f7212ba7a61553c47c828420e98164c"
server_key_version: "encryptionkey@0"
I20260927 00:56:57.272547 21197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:56:57.275496 21212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:57.277187 21197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:56:57.277343 21197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "85d4826d624c4290a478666406b1f2b9"
format_stamp: "Formatted at 2026-09-27 00:56:57 on dist-test-slave-9fh2"
server_key: "2b2bc2e660aa4a98dcb41abad9cc27b1"
server_key_iv: "9f7212ba7a61553c47c828420e98164c"
server_key_version: "encryptionkey@0"
I20260927 00:56:57.277464 21197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:57.282248 21197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:57.286525 21197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:57.286755 21197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:57.294752 21197 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46215
I20260927 00:56:57.294884 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46215 every 8 connection(s)
I20260927 00:56:57.295830 21197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:57.299010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21197
I20260927 00:56:57.299211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:57.299571 392 external_mini_cluster.cc:1428] Setting key 0101e8cc4a8060b2f69e3090f3e60d9b
I20260927 00:56:57.299721 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:57.306391 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:57.301426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36523 (local address 127.0.0.1:46215)
0927 00:56:57.301937 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:57.301954 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:57.301996 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:56:57.303079 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:57.303101 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:57.303511 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:57.303883 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:57.304980 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:57.304985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:57.304998 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:57.305015 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:57.305107 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:57.305129 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:57.305273 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:57.305365 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 00:56:57.305518 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":153,"threads_started":1}
I20260927 00:56:57.308563 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9: Bootstrap starting.
I20260927 00:56:57.310082 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:56:57.311879 21265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:57.313005 21265 log.cc:824] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:56:57.314983 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:57.313061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52075 (local address 127.0.0.1:46215)
0927 00:56:57.313229 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:57.313234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:57.313462 (+ 228us) server_negotiation.cc:407] Connection header received
0927 00:56:57.313638 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:57.313641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:57.313693 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:57.313801 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:57.314605 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:57.314608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:57.314610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:56:57.314621 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:57.314654 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:56:57.314665 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:57.314752 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:57.314832 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 00:56:57.314868 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:56:57.316313 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9: No bootstrap required, opened a new log
I20260927 00:56:57.320844 21265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4826d624c4290a478666406b1f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46215 } }
I20260927 00:56:57.321208 21265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:57.321249 21265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d4826d624c4290a478666406b1f2b9, State: Initialized, Role: FOLLOWER
I20260927 00:56:57.321777 21265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4826d624c4290a478666406b1f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46215 } }
I20260927 00:56:57.321925 21265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:57.321976 21265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:57.322088 21265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:57.323436 21265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4826d624c4290a478666406b1f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46215 } }
I20260927 00:56:57.323920 21265 leader_election.cc:302] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85d4826d624c4290a478666406b1f2b9; no voters:
I20260927 00:56:57.324390 21265 leader_election.cc:288] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:57.324580 21275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:57.324958 21275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [term 1 LEADER]: Becoming Leader. State: Replica: 85d4826d624c4290a478666406b1f2b9, State: Running, Role: LEADER
I20260927 00:56:57.325380 21275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4826d624c4290a478666406b1f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46215 } }
I20260927 00:56:57.325980 21265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:57.327332 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85d4826d624c4290a478666406b1f2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4826d624c4290a478666406b1f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46215 } } }
I20260927 00:56:57.327507 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:57.327946 21284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:57.327934 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85d4826d624c4290a478666406b1f2b9. Latest consensus state: current_term: 1 leader_uuid: "85d4826d624c4290a478666406b1f2b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d4826d624c4290a478666406b1f2b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46215 } } }
I20260927 00:56:57.328606 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:57.332233 21284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:57.339593 21284 catalog_manager.cc:1347] Generated new cluster ID: a1c92b70f45f41a2ab37b9934ff3a65f
I20260927 00:56:57.339717 21284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:57.353823 21284 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:57.354324 21284 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:57.361837 21284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85d4826d624c4290a478666406b1f2b9: Generated new TSK 0
I20260927 00:56:57.363026 21284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:57.371232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21197
2026-09-27T00:56:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:57 dist-test-slave-9fh2 krb5kdc[21296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:57 dist-test-slave-9fh2 krb5kdc[21296](info): set up 2 sockets
Sep 27 00:56:57 dist-test-slave-9fh2 krb5kdc[21296](info): commencing operation
krb5kdc: starting...
W20260927 00:56:58.339656 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:58 dist-test-slave-9fh2 krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:58.479619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:58.589365 21309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:58.589798 21309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:58.589856 21309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:58.595042 21309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:58.595202 21309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:58.595228 21309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:58.595250 21309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:58.595293 21309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:58.602434 21309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:58.604023 21309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:58.605281 21309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:58.615482 21315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:58.615530 21314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:58.615658 21309 server_base.cc:1034] running on GCE node
W20260927 00:56:58.615484 21317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:58.616523 21309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:58.617692 21309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:58.618899 21309 hybrid_clock.cc:648] HybridClock initialized: now 1790470618618874 us; error 42 us; skew 500 ppm
Sep 27 00:56:58 dist-test-slave-9fh2 krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:58.621981 21309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:58.622952 21309 webserver.cc:460] Webserver started at http://127.0.0.1:36069/ using document root <none> and password file <none>
I20260927 00:56:58.623553 21309 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:58.623627 21309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:58.623809 21309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:58.625821 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "1de90f8e7b874d2b83a13d14c5b54cda"
format_stamp: "Formatted at 2026-09-27 00:56:58 on dist-test-slave-9fh2"
server_key: "131b16a5db1e3c3fe330370e7bbfc299"
server_key_iv: "4b8e8399f02daebedc1cce55bf64dd41"
server_key_version: "encryptionkey@0"
I20260927 00:56:58.626336 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "1de90f8e7b874d2b83a13d14c5b54cda"
format_stamp: "Formatted at 2026-09-27 00:56:58 on dist-test-slave-9fh2"
server_key: "131b16a5db1e3c3fe330370e7bbfc299"
server_key_iv: "4b8e8399f02daebedc1cce55bf64dd41"
server_key_version: "encryptionkey@0"
I20260927 00:56:58.630430 21309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 00:56:58.633334 21324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:56:58.634896 21309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:56:58.635037 21309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "1de90f8e7b874d2b83a13d14c5b54cda"
format_stamp: "Formatted at 2026-09-27 00:56:58 on dist-test-slave-9fh2"
server_key: "131b16a5db1e3c3fe330370e7bbfc299"
server_key_iv: "4b8e8399f02daebedc1cce55bf64dd41"
server_key_version: "encryptionkey@0"
I20260927 00:56:58.635197 21309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:56:58.658420 21309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:56:58.661968 21309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:56:58.662169 21309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:56:58.669502 21309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44015
I20260927 00:56:58.669554 21376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44015 every 8 connection(s)
I20260927 00:56:58.670514 21309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:56:58.674289 21377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:56:58.677508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21309
I20260927 00:56:58.677698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:56:58.677984 392 external_mini_cluster.cc:1428] Setting key 39313c8ff1341615c91a1d245195e8b3
I20260927 00:56:58.682518 21377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda: Bootstrap starting.
I20260927 00:56:58.685945 21377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda: Neither blocks nor log segments found. Creating new log.
I20260927 00:56:58.686882 21377 log.cc:824] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:56:58 dist-test-slave-9fh2 krb5kdc[21296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:56:58.689785 21377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda: No bootstrap required, opened a new log
I20260927 00:56:58.693473 21377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44015 } }
I20260927 00:56:58.693812 21377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:56:58.693893 21377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de90f8e7b874d2b83a13d14c5b54cda, State: Initialized, Role: FOLLOWER
I20260927 00:56:58.694430 21377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44015 } }
I20260927 00:56:58.694589 21377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:56:58.694639 21377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:56:58.694754 21377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:56:58.695885 21377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44015 } }
I20260927 00:56:58.696271 21377 leader_election.cc:302] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de90f8e7b874d2b83a13d14c5b54cda; no voters:
I20260927 00:56:58.696620 21377 leader_election.cc:288] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:56:58.697096 21383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:56:58.697431 21383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [term 1 LEADER]: Becoming Leader. State: Replica: 1de90f8e7b874d2b83a13d14c5b54cda, State: Running, Role: LEADER
I20260927 00:56:58.697875 21383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44015 } }
I20260927 00:56:58.698104 21377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:56:58.698197 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:58.679907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:34223 (local address 127.0.0.1:44015)
0927 00:56:58.680467 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:58.680490 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:58.680548 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:56:58.681542 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:58.681573 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:58.682163 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:58.682626 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:58.683507 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:58.685115 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:58.686227 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:58.686888 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:58.687086 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:58.690451 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:58.690459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:58.690481 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:58.690529 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:58.693449 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:58.694355 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:58.694362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:58.694368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:58.694662 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:58.695137 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:58.695140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:58.695177 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:58.695770 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:58.696047 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:58.696437 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 00:56:58.696768 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":146,"threads_started":1}
I20260927 00:56:58.699988 21382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44015 } } }
I20260927 00:56:58.700155 21382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:58.700802 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de90f8e7b874d2b83a13d14c5b54cda. Latest consensus state: current_term: 1 leader_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de90f8e7b874d2b83a13d14c5b54cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44015 } } }
I20260927 00:56:58.700922 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda [sys.catalog]: This master's current role is: LEADER
I20260927 00:56:58.702525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 00:56:58.703892 21396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:56:58.704007 21396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:56:58.704087 21397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:56:58.705199 21397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:56:58.713913 21397 catalog_manager.cc:1347] Generated new cluster ID: 64d54c5df1d54ad89441884a32ba8265
I20260927 00:56:58.714023 21397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:56:58.715838 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:56:58.704958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:37505 (local address 127.0.0.1:44015)
0927 00:56:58.705073 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:56:58.705078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:56:58.705474 (+ 396us) server_negotiation.cc:407] Connection header received
0927 00:56:58.705656 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:56:58.705659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:56:58.705717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:56:58.705771 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:56:58.706707 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:58.707576 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:58.708755 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:56:58.709119 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:56:58.709194 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:56:58.711135 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:56:58.711140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:56:58.711166 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:56:58.711194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:56:58.713416 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:58.714146 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:58.714150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:58.714152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:58.714257 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:56:58.714657 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:56:58.714661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:56:58.714662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:56:58.714891 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:56:58.715030 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:56:58.715541 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 00:56:58.715657 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:56:58.721845 21397 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:56:58.723263 21397 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:56:58.731594 21397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de90f8e7b874d2b83a13d14c5b54cda: Generated new TSK 0
I20260927 00:56:58.732458 21397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:56:58.745965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21309
W20260927 00:56:58.751083 21294 connection.cc:535] client connection to 127.0.0.1:44015 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44015): Connection reset by peer (error 104)
2026-09-27T00:56:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:56:58 dist-test-slave-9fh2 krb5kdc[21408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:56:58 dist-test-slave-9fh2 krb5kdc[21408](info): set up 2 sockets
Sep 27 00:56:58 dist-test-slave-9fh2 krb5kdc[21408](info): commencing operation
krb5kdc: starting...
W20260927 00:56:59.711663 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:56:59 dist-test-slave-9fh2 krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:56:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:56:59.850584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:56:59.959089 21421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:56:59.959404 21421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:56:59.959447 21421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:56:59.963507 21421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:56:59.963582 21421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:56:59.963601 21421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:56:59.963620 21421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:56:59.963635 21421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:56:59.969247 21421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:56:59.970377 21421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:56:59.971422 21421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:56:59.980556 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:59.980516 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:56:59.980541 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:56:59.981048 21421 server_base.cc:1034] running on GCE node
I20260927 00:56:59.981793 21421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:56:59.983289 21421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:56:59.984508 21421 hybrid_clock.cc:648] HybridClock initialized: now 1790470619984486 us; error 41 us; skew 500 ppm
Sep 27 00:56:59 dist-test-slave-9fh2 krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:56:59.988046 21421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:56:59.989106 21421 webserver.cc:460] Webserver started at http://127.0.0.1:33679/ using document root <none> and password file <none>
I20260927 00:56:59.989646 21421 fs_manager.cc:359] Metadata directory not provided
I20260927 00:56:59.989712 21421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:56:59.989880 21421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:56:59.991964 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c556a4880aa54ca3b94e12f58c67e21f"
format_stamp: "Formatted at 2026-09-27 00:56:59 on dist-test-slave-9fh2"
server_key: "b28e14d154f409392ccb45d45ae2ae16"
server_key_iv: "2b079da9d26cae47a2b1c59c866456a5"
server_key_version: "encryptionkey@0"
I20260927 00:56:59.992481 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c556a4880aa54ca3b94e12f58c67e21f"
format_stamp: "Formatted at 2026-09-27 00:56:59 on dist-test-slave-9fh2"
server_key: "b28e14d154f409392ccb45d45ae2ae16"
server_key_iv: "2b079da9d26cae47a2b1c59c866456a5"
server_key_version: "encryptionkey@0"
I20260927 00:56:59.996510 21421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:56:59.999380 21436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:00.000998 21421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:57:00.001184 21421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c556a4880aa54ca3b94e12f58c67e21f"
format_stamp: "Formatted at 2026-09-27 00:56:59 on dist-test-slave-9fh2"
server_key: "b28e14d154f409392ccb45d45ae2ae16"
server_key_iv: "2b079da9d26cae47a2b1c59c866456a5"
server_key_version: "encryptionkey@0"
I20260927 00:57:00.001312 21421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:00.015367 21421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:00.018507 21421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:00.018710 21421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:00.026064 21421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35223
I20260927 00:57:00.026163 21488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35223 every 8 connection(s)
I20260927 00:57:00.027093 21421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:00.028533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21421
I20260927 00:57:00.028699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:00.029045 392 external_mini_cluster.cc:1428] Setting key 98a43efb7ede231306e16ffe70c8843c
I20260927 00:57:00.031345 21489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:00.038651 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:00.030738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:59977 (local address 127.0.0.1:35223)
0927 00:57:00.031356 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:00.031375 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:00.031420 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:57:00.032127 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:00.032146 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:00.032534 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:00.032897 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:00.033329 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:00.034364 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:00.035317 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:00.035833 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:00.036000 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:00.036887 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:00.036892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:00.036906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:00.037334 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:00.037392 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:00.037415 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:00.037548 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:00.037750 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 00:57:00.037891 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":184,"threads_started":1}
I20260927 00:57:00.038792 21489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f: Bootstrap starting.
I20260927 00:57:00.042140 21489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:00.042358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:00.043267 21489 log.cc:824] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:00.046097 21489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f: No bootstrap required, opened a new log
I20260927 00:57:00.049451 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:00.045088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:51995 (local address 127.0.0.1:35223)
0927 00:57:00.045232 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:00.045236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:00.045333 (+ 97us) server_negotiation.cc:407] Connection header received
0927 00:57:00.045445 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:00.045448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:00.045502 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:00.045619 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:00.045840 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:00.046633 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:00.047579 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:00.047930 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:00.048077 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:00.048881 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:00.048885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:00.048888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:00.048904 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:00.048934 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:00.048948 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:00.049044 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:00.049226 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:57:00.049272 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:57:00.051079 21489 raft_consensus.cc:348] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c556a4880aa54ca3b94e12f58c67e21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260927 00:57:00.051406 21489 raft_consensus.cc:374] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:00.051462 21489 raft_consensus.cc:729] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c556a4880aa54ca3b94e12f58c67e21f, State: Initialized, Role: FOLLOWER
I20260927 00:57:00.051990 21489 consensus_queue.cc:260] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c556a4880aa54ca3b94e12f58c67e21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260927 00:57:00.052130 21489 raft_consensus.cc:388] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:00.052181 21489 raft_consensus.cc:482] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:00.052290 21489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:00.053382 21489 raft_consensus.cc:504] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c556a4880aa54ca3b94e12f58c67e21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260927 00:57:00.053813 21489 leader_election.cc:302] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c556a4880aa54ca3b94e12f58c67e21f; no voters:
I20260927 00:57:00.054289 21489 leader_election.cc:288] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:00.054423 21499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:00.054702 21499 raft_consensus.cc:686] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [term 1 LEADER]: Becoming Leader. State: Replica: c556a4880aa54ca3b94e12f58c67e21f, State: Running, Role: LEADER
I20260927 00:57:00.055135 21499 consensus_queue.cc:237] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c556a4880aa54ca3b94e12f58c67e21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260927 00:57:00.055881 21489 sys_catalog.cc:564] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:00.057013 21500 sys_catalog.cc:455] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c556a4880aa54ca3b94e12f58c67e21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c556a4880aa54ca3b94e12f58c67e21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } } }
I20260927 00:57:00.057183 21500 sys_catalog.cc:458] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:00.057598 21501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c556a4880aa54ca3b94e12f58c67e21f. Latest consensus state: current_term: 1 leader_uuid: "c556a4880aa54ca3b94e12f58c67e21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c556a4880aa54ca3b94e12f58c67e21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } } }
I20260927 00:57:00.057679 21508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:00.057703 21501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:00.061344 21508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:00.069427 21508 catalog_manager.cc:1347] Generated new cluster ID: 915ca55bfc654919b9cf76beabfda09d
I20260927 00:57:00.069506 21508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:00.081734 21508 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:00.083268 21508 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:00.099541 21508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c556a4880aa54ca3b94e12f58c67e21f: Generated new TSK 0
I20260927 00:57:00.100751 21508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:00.112123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21421
W20260927 00:57:00.117367 21406 connection.cc:535] client connection to 127.0.0.1:35223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35223): Connection reset by peer (error 104)
2026-09-27T00:57:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:00 dist-test-slave-9fh2 krb5kdc[21520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:00 dist-test-slave-9fh2 krb5kdc[21520](info): set up 2 sockets
Sep 27 00:57:00 dist-test-slave-9fh2 krb5kdc[21520](info): commencing operation
krb5kdc: starting...
W20260927 00:57:01.079941 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:01 dist-test-slave-9fh2 krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:57:01.214366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:01.345922 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:01.346426 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:01.346477 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:01.351030 21533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:01.351178 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:01.351212 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:01.351238 21533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:01.351260 21533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:01.357640 21533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:01.359083 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:01.360380 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:01.370543 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:01.370514 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:01.370496 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:01.371047 21533 server_base.cc:1034] running on GCE node
I20260927 00:57:01.371853 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:01.373297 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:01.374491 21533 hybrid_clock.cc:648] HybridClock initialized: now 1790470621374464 us; error 48 us; skew 500 ppm
Sep 27 00:57:01 dist-test-slave-9fh2 krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:01.378118 21533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:57:01.379407 21533 webserver.cc:460] Webserver started at http://127.0.0.1:39289/ using document root <none> and password file <none>
I20260927 00:57:01.380122 21533 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:01.380209 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:01.380448 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:01.383030 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "c611c8c5d95f47c184879afe8060cffd"
format_stamp: "Formatted at 2026-09-27 00:57:01 on dist-test-slave-9fh2"
server_key: "0a3a84cad4510d221ee96b8fa644977a"
server_key_iv: "344dad00f624db4e1f2603062d0a2278"
server_key_version: "encryptionkey@0"
I20260927 00:57:01.383679 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "c611c8c5d95f47c184879afe8060cffd"
format_stamp: "Formatted at 2026-09-27 00:57:01 on dist-test-slave-9fh2"
server_key: "0a3a84cad4510d221ee96b8fa644977a"
server_key_iv: "344dad00f624db4e1f2603062d0a2278"
server_key_version: "encryptionkey@0"
I20260927 00:57:01.388506 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:57:01.391728 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:01.393420 21533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:57:01.393596 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "c611c8c5d95f47c184879afe8060cffd"
format_stamp: "Formatted at 2026-09-27 00:57:01 on dist-test-slave-9fh2"
server_key: "0a3a84cad4510d221ee96b8fa644977a"
server_key_iv: "344dad00f624db4e1f2603062d0a2278"
server_key_version: "encryptionkey@0"
I20260927 00:57:01.393738 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:01.399298 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:01.403424 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:01.403738 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:01.412952 21533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34531
I20260927 00:57:01.412992 21600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34531 every 8 connection(s)
I20260927 00:57:01.414186 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:01.417387 21601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:01.422676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21533
I20260927 00:57:01.422832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:01.423135 392 external_mini_cluster.cc:1428] Setting key 2010aee0fe7b270834c341a58c6ebd50
I20260927 00:57:01.423503 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd: Bootstrap starting.
I20260927 00:57:01.426098 21601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:01.426859 21601 log.cc:824] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:01.428941 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd: No bootstrap required, opened a new log
I20260927 00:57:01.429927 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:01.424817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:53901 (local address 127.0.0.1:34531)
0927 00:57:01.425447 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:01.425476 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:01.425528 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:57:01.426525 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:01.426555 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:01.427005 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:01.427465 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:01.428392 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:01.428398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:01.428412 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:01.428424 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:01.428488 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:01.428510 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:01.428621 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:01.428730 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:57:01.428937 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":171,"threads_started":1}
I20260927 00:57:01.432878 21601 raft_consensus.cc:348] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c611c8c5d95f47c184879afe8060cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260927 00:57:01.433136 21601 raft_consensus.cc:374] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:01.433178 21601 raft_consensus.cc:729] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c611c8c5d95f47c184879afe8060cffd, State: Initialized, Role: FOLLOWER
I20260927 00:57:01.433508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:01.433640 21601 consensus_queue.cc:260] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c611c8c5d95f47c184879afe8060cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260927 00:57:01.433790 21601 raft_consensus.cc:388] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:01.433841 21601 raft_consensus.cc:482] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:01.433934 21601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:01.435201 21601 raft_consensus.cc:504] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c611c8c5d95f47c184879afe8060cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260927 00:57:01.435613 21601 leader_election.cc:302] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c611c8c5d95f47c184879afe8060cffd; no voters:
I20260927 00:57:01.435961 21601 leader_election.cc:288] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:01.436054 21610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:01.436232 21610 raft_consensus.cc:686] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [term 1 LEADER]: Becoming Leader. State: Replica: c611c8c5d95f47c184879afe8060cffd, State: Running, Role: LEADER
I20260927 00:57:01.436523 21610 consensus_queue.cc:237] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c611c8c5d95f47c184879afe8060cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } }
I20260927 00:57:01.437294 21601 sys_catalog.cc:564] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:01.438148 21610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c611c8c5d95f47c184879afe8060cffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c611c8c5d95f47c184879afe8060cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } } }
I20260927 00:57:01.438423 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:01.436222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:37817 (local address 127.0.0.1:34531)
0927 00:57:01.436349 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:01.436354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:01.436731 (+ 377us) server_negotiation.cc:407] Connection header received
0927 00:57:01.436916 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:01.436919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:01.436964 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:01.437092 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:01.437868 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:01.437872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:01.437875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:01.437888 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:01.437918 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:01.437929 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:01.438006 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:01.438163 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:57:01.438212 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:57:01.438277 21610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:01.438858 21610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c611c8c5d95f47c184879afe8060cffd. Latest consensus state: current_term: 1 leader_uuid: "c611c8c5d95f47c184879afe8060cffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c611c8c5d95f47c184879afe8060cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34531 } } }
I20260927 00:57:01.438951 21610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:01.439767 21620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:01.443260 21620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:01.449699 21620 catalog_manager.cc:1347] Generated new cluster ID: de58c23e254c4e47a98c35d9f3638db9
I20260927 00:57:01.449801 21620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:01.477793 21620 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:01.478189 21620 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:01.483031 21620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c611c8c5d95f47c184879afe8060cffd: Generated new TSK 0
I20260927 00:57:01.484099 21620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:01.489276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21533
2026-09-27T00:57:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:01 dist-test-slave-9fh2 krb5kdc[21632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:01 dist-test-slave-9fh2 krb5kdc[21632](info): set up 2 sockets
Sep 27 00:57:01 dist-test-slave-9fh2 krb5kdc[21632](info): commencing operation
krb5kdc: starting...
W20260927 00:57:02.455808 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:02 dist-test-slave-9fh2 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:57:02.591379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:02.723503 21645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:02.723878 21645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:02.723944 21645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:02.729449 21645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:02.729549 21645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:02.729578 21645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:02.729600 21645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:02.729624 21645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:02.736086 21645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:02.737634 21645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:02.738901 21645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:02.749202 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:02.749186 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:02.749208 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:02.749671 21645 server_base.cc:1034] running on GCE node
I20260927 00:57:02.750324 21645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:02.751624 21645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:02.752800 21645 hybrid_clock.cc:648] HybridClock initialized: now 1790470622752775 us; error 48 us; skew 500 ppm
Sep 27 00:57:02 dist-test-slave-9fh2 krb5kdc[21632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:02.759438 21645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:57:02.760545 21645 webserver.cc:460] Webserver started at http://127.0.0.1:32925/ using document root <none> and password file <none>
I20260927 00:57:02.761164 21645 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:02.761243 21645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:02.761461 21645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:02.763774 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa"
format_stamp: "Formatted at 2026-09-27 00:57:02 on dist-test-slave-9fh2"
server_key: "d01030ef6589e487da4902a4713b3ce9"
server_key_iv: "0c2c96783a82022b070d9fadc75d40ee"
server_key_version: "encryptionkey@0"
I20260927 00:57:02.764349 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa"
format_stamp: "Formatted at 2026-09-27 00:57:02 on dist-test-slave-9fh2"
server_key: "d01030ef6589e487da4902a4713b3ce9"
server_key_iv: "0c2c96783a82022b070d9fadc75d40ee"
server_key_version: "encryptionkey@0"
I20260927 00:57:02.768771 21645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:57:02.771795 21660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:02.773345 21645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:02.773495 21645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa"
format_stamp: "Formatted at 2026-09-27 00:57:02 on dist-test-slave-9fh2"
server_key: "d01030ef6589e487da4902a4713b3ce9"
server_key_iv: "0c2c96783a82022b070d9fadc75d40ee"
server_key_version: "encryptionkey@0"
I20260927 00:57:02.773607 21645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:02.794740 21645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:02.798177 21645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:02.798384 21645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:02.806324 21645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33151
I20260927 00:57:02.806420 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33151 every 8 connection(s)
I20260927 00:57:02.807474 21645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:02.810034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21645
I20260927 00:57:02.810184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:02.810464 392 external_mini_cluster.cc:1428] Setting key fa3a1ac54fa3ceadf063288e5b1116c3
I20260927 00:57:02.811810 21713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:02.819429 21713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa: Bootstrap starting.
Sep 27 00:57:02 dist-test-slave-9fh2 krb5kdc[21632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:57:02.822520 21713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:02.823474 21713 log.cc:824] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:02.826179 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:02.812187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:44163 (local address 127.0.0.1:33151)
0927 00:57:02.812841 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:02.812861 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:02.812909 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:57:02.813646 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:02.813668 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:02.814069 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:02.814418 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:02.815265 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:02.816313 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:02.817176 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:02.817706 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:02.817865 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:02.820849 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:02.820855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:02.820871 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:02.820905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:02.822765 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:02.823316 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:02.823321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:02.823326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:02.823448 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:02.823771 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:02.823774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:02.823775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:02.824317 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:02.824526 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:02.825083 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 00:57:02.825336 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":207,"threads_started":1}
I20260927 00:57:02.826367 21713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa: No bootstrap required, opened a new log
I20260927 00:57:02.830057 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:02.831663 21713 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260927 00:57:02.832079 21713 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:02.832147 21713 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7c2a8e7bca4d188f726e82e0ba1baa, State: Initialized, Role: FOLLOWER
I20260927 00:57:02.832715 21713 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260927 00:57:02.832890 21713 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:02.832959 21713 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:02.833063 21713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:02.834481 21713 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260927 00:57:02.834998 21713 leader_election.cc:302] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7c2a8e7bca4d188f726e82e0ba1baa; no voters:
I20260927 00:57:02.835477 21713 leader_election.cc:288] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:02.835649 21723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:02.835968 21723 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [term 1 LEADER]: Becoming Leader. State: Replica: ad7c2a8e7bca4d188f726e82e0ba1baa, State: Running, Role: LEADER
I20260927 00:57:02.836414 21723 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260927 00:57:02.837178 21713 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:02.838387 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260927 00:57:02.838559 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:02.838878 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7c2a8e7bca4d188f726e82e0ba1baa. Latest consensus state: current_term: 1 leader_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c2a8e7bca4d188f726e82e0ba1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260927 00:57:02.838964 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:02.839350 21732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:02.842460 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:02.832827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:55011 (local address 127.0.0.1:33151)
0927 00:57:02.832981 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:02.832986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:02.833198 (+ 212us) server_negotiation.cc:407] Connection header received
0927 00:57:02.833336 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:02.833339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:02.833409 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:02.833546 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:02.834352 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:02.835227 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:02.836334 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:02.836674 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:02.836791 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:02.838767 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:02.838772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:02.838775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:02.838811 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:02.840471 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:02.841007 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:02.841010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:02.841012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:02.841126 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:02.841551 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:02.841554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:02.841556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:02.841761 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:02.841902 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:02.842177 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:57:02.842287 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:57:02.843319 21732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:02.850001 21732 catalog_manager.cc:1347] Generated new cluster ID: 7a9a64bed8b04b37abe014348b7f3925
I20260927 00:57:02.850082 21732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:02.872005 21732 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:02.873301 21732 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:02.879540 21732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad7c2a8e7bca4d188f726e82e0ba1baa: Generated new TSK 0
I20260927 00:57:02.880548 21732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:02.892863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21645
W20260927 00:57:02.899377 21630 connection.cc:535] client connection to 127.0.0.1:33151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33151): Connection reset by peer (error 104)
2026-09-27T00:57:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:02 dist-test-slave-9fh2 krb5kdc[21744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:02 dist-test-slave-9fh2 krb5kdc[21744](info): set up 2 sockets
Sep 27 00:57:02 dist-test-slave-9fh2 krb5kdc[21744](info): commencing operation
krb5kdc: starting...
W20260927 00:57:03.859701 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:03 dist-test-slave-9fh2 krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:57:03.998742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:04.116624 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:04.116930 21757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:04.116971 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:04.121126 21757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:04.121204 21757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:04.121222 21757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:04.121243 21757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:04.121258 21757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:04.126811 21757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:04.128027 21757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:04.129117 21757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:04.138433 21762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:04.138633 21757 server_base.cc:1034] running on GCE node
W20260927 00:57:04.138447 21763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:04.138433 21765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:04.139545 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:04.140705 21757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:04.141916 21757 hybrid_clock.cc:648] HybridClock initialized: now 1790470624141886 us; error 56 us; skew 500 ppm
Sep 27 00:57:04 dist-test-slave-9fh2 krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:04.144892 21757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:57:04.145910 21757 webserver.cc:460] Webserver started at http://127.0.0.1:43837/ using document root <none> and password file <none>
I20260927 00:57:04.146509 21757 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:04.146581 21757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:04.146755 21757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:04.148883 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "1f36bbef7ee4433e8805d21779aaa068"
format_stamp: "Formatted at 2026-09-27 00:57:04 on dist-test-slave-9fh2"
server_key: "f71f5b86624645b93b85b9f515303c85"
server_key_iv: "60f99fee64d8d72a3694e635003e6a0f"
server_key_version: "encryptionkey@0"
I20260927 00:57:04.149433 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f36bbef7ee4433e8805d21779aaa068"
format_stamp: "Formatted at 2026-09-27 00:57:04 on dist-test-slave-9fh2"
server_key: "f71f5b86624645b93b85b9f515303c85"
server_key_iv: "60f99fee64d8d72a3694e635003e6a0f"
server_key_version: "encryptionkey@0"
I20260927 00:57:04.153689 21757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:57:04.156689 21772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:04.158293 21757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:04.158432 21757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "1f36bbef7ee4433e8805d21779aaa068"
format_stamp: "Formatted at 2026-09-27 00:57:04 on dist-test-slave-9fh2"
server_key: "f71f5b86624645b93b85b9f515303c85"
server_key_iv: "60f99fee64d8d72a3694e635003e6a0f"
server_key_version: "encryptionkey@0"
I20260927 00:57:04.158545 21757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:04.176717 21757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:04.179910 21757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:04.180073 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:04.187849 21757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40329
I20260927 00:57:04.187860 21824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40329 every 8 connection(s)
I20260927 00:57:04.188908 21757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:04.192776 21825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:04.197224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21757
I20260927 00:57:04.197366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:04.197681 392 external_mini_cluster.cc:1428] Setting key dd3571ac486c6f9311af93df3f1a16af
I20260927 00:57:04.200569 21825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068: Bootstrap starting.
I20260927 00:57:04.203791 21825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:04.204800 21825 log.cc:824] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:04.207841 21825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068: No bootstrap required, opened a new log
I20260927 00:57:04.209081 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:04.199255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:60263 (local address 127.0.0.1:40329)
0927 00:57:04.199724 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:04.199740 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:04.199780 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:57:04.200565 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:04.200591 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:04.201001 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:04.201323 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:04.201868 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:04.203057 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:04.204421 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:04.205075 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:04.205221 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:04.206481 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:04.206502 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:04.206524 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:04.207182 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:04.207298 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:04.207338 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:04.207521 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:04.207746 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 00:57:04.207927 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":128,"threads_started":1}
I20260927 00:57:04.211925 21825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36bbef7ee4433e8805d21779aaa068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260927 00:57:04.212282 21825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:04.212335 21825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f36bbef7ee4433e8805d21779aaa068, State: Initialized, Role: FOLLOWER
I20260927 00:57:04.212906 21825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36bbef7ee4433e8805d21779aaa068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260927 00:57:04.213083 21825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:04.213156 21825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:04.213258 21825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:04.213346 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:04.214676 21825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36bbef7ee4433e8805d21779aaa068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260927 00:57:04.215117 21825 leader_election.cc:302] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f36bbef7ee4433e8805d21779aaa068; no voters:
I20260927 00:57:04.215556 21825 leader_election.cc:288] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:04.215710 21834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:04.215934 21834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [term 1 LEADER]: Becoming Leader. State: Replica: 1f36bbef7ee4433e8805d21779aaa068, State: Running, Role: LEADER
I20260927 00:57:04.216316 21834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36bbef7ee4433e8805d21779aaa068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260927 00:57:04.217191 21825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:04.218287 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f36bbef7ee4433e8805d21779aaa068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36bbef7ee4433e8805d21779aaa068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } } }
I20260927 00:57:04.218466 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:04.218943 21844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:04.219100 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f36bbef7ee4433e8805d21779aaa068. Latest consensus state: current_term: 1 leader_uuid: "1f36bbef7ee4433e8805d21779aaa068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36bbef7ee4433e8805d21779aaa068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } } }
I20260927 00:57:04.219223 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:04.220669 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:04.215501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:42613 (local address 127.0.0.1:40329)
0927 00:57:04.215646 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:04.215651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:04.215864 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:57:04.216075 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:04.216078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:04.216134 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:04.216187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:04.216589 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:04.217153 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:04.218168 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:04.218500 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:04.218595 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:04.219880 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:04.219885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:04.219887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:04.219902 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:04.219931 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:04.219946 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:04.220019 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:04.220470 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 00:57:04.220511 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260927 00:57:04.223783 21844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:04.231396 21844 catalog_manager.cc:1347] Generated new cluster ID: 4ec39537099e4ad68f00a12fa2a26732
I20260927 00:57:04.231487 21844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:04.240375 21844 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:04.241559 21844 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:04.247735 21844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f36bbef7ee4433e8805d21779aaa068: Generated new TSK 0
I20260927 00:57:04.248674 21844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:04.262260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21757
W20260927 00:57:04.267657 21742 connection.cc:535] client connection to 127.0.0.1:40329 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40329): Connection reset by peer (error 104)
2026-09-27T00:57:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:04 dist-test-slave-9fh2 krb5kdc[21856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:04 dist-test-slave-9fh2 krb5kdc[21856](info): set up 2 sockets
Sep 27 00:57:04 dist-test-slave-9fh2 krb5kdc[21856](info): commencing operation
krb5kdc: starting...
W20260927 00:57:05.243656 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:05 dist-test-slave-9fh2 krb5kdc[21856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:57:05.379653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:05.510672 21869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:05.511076 21869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:05.511175 21869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:05.515733 21869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:05.515851 21869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:05.515882 21869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:05.515905 21869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:05.515928 21869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:05.522311 21869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:05.523922 21869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:05.525234 21869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:05.535820 21874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:05.535820 21877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:05.535857 21875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:05.536437 21869 server_base.cc:1034] running on GCE node
I20260927 00:57:05.537202 21869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:05.538769 21869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:05.539984 21869 hybrid_clock.cc:648] HybridClock initialized: now 1790470625539949 us; error 65 us; skew 500 ppm
Sep 27 00:57:05 dist-test-slave-9fh2 krb5kdc[21856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:05.543524 21869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:57:05.545176 21869 webserver.cc:460] Webserver started at http://127.0.0.1:34105/ using document root <none> and password file <none>
I20260927 00:57:05.546185 21869 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:05.546288 21869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:05.546577 21869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:05.549974 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "2afc3b185faa4de38d8264765cad8bf2"
format_stamp: "Formatted at 2026-09-27 00:57:05 on dist-test-slave-9fh2"
server_key: "022e5abdf04a419900abac87d34f421b"
server_key_iv: "dff0ba3b9ce4603e8417741a678a65aa"
server_key_version: "encryptionkey@0"
I20260927 00:57:05.550758 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "2afc3b185faa4de38d8264765cad8bf2"
format_stamp: "Formatted at 2026-09-27 00:57:05 on dist-test-slave-9fh2"
server_key: "022e5abdf04a419900abac87d34f421b"
server_key_iv: "dff0ba3b9ce4603e8417741a678a65aa"
server_key_version: "encryptionkey@0"
I20260927 00:57:05.555636 21869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 00:57:05.558888 21884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:05.560631 21869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:05.560807 21869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "2afc3b185faa4de38d8264765cad8bf2"
format_stamp: "Formatted at 2026-09-27 00:57:05 on dist-test-slave-9fh2"
server_key: "022e5abdf04a419900abac87d34f421b"
server_key_iv: "dff0ba3b9ce4603e8417741a678a65aa"
server_key_version: "encryptionkey@0"
I20260927 00:57:05.560949 21869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:05.566569 21869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:05.570690 21869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:05.570920 21869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:05.581003 21869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38169
I20260927 00:57:05.581033 21936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38169 every 8 connection(s)
I20260927 00:57:05.582458 21869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:05.585858 21937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:05.587502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21869
I20260927 00:57:05.587651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:05.587977 392 external_mini_cluster.cc:1428] Setting key 28047097da606bb32a8186adf9656831
I20260927 00:57:05.593230 21937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2: Bootstrap starting.
I20260927 00:57:05.594480 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:05.589681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:52095 (local address 127.0.0.1:38169)
0927 00:57:05.590230 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:05.590250 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:05.590294 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:57:05.591129 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:05.591191 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:05.591555 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:05.591943 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:05.592958 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:05.592963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:05.592979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:05.592992 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:05.593067 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:05.593095 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:05.593272 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:05.593406 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:57:05.593562 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":162,"threads_started":1}
I20260927 00:57:05.595906 21937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:05.596657 21937 log.cc:824] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:05.598073 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:05.598884 21937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2: No bootstrap required, opened a new log
I20260927 00:57:05.602664 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:05.600715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:55643 (local address 127.0.0.1:38169)
0927 00:57:05.600868 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:05.600873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:05.601133 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:57:05.601257 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:05.601261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:05.601299 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:05.601395 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:05.602231 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:05.602234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:05.602237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:05.602249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:05.602277 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:05.602290 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:05.602387 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:05.602490 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 00:57:05.602533 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:57:05.602811 21937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc3b185faa4de38d8264765cad8bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 00:57:05.603070 21937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:05.603132 21937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afc3b185faa4de38d8264765cad8bf2, State: Initialized, Role: FOLLOWER
I20260927 00:57:05.603601 21937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc3b185faa4de38d8264765cad8bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 00:57:05.603729 21937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:05.603791 21937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:05.603890 21937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:05.604825 21937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc3b185faa4de38d8264765cad8bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 00:57:05.605178 21937 leader_election.cc:302] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2afc3b185faa4de38d8264765cad8bf2; no voters:
I20260927 00:57:05.605484 21937 leader_election.cc:288] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:05.605684 21947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:05.606031 21947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 2afc3b185faa4de38d8264765cad8bf2, State: Running, Role: LEADER
I20260927 00:57:05.606449 21947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc3b185faa4de38d8264765cad8bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 00:57:05.606806 21937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:05.608721 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2afc3b185faa4de38d8264765cad8bf2. Latest consensus state: current_term: 1 leader_uuid: "2afc3b185faa4de38d8264765cad8bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc3b185faa4de38d8264765cad8bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260927 00:57:05.608779 21948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2afc3b185faa4de38d8264765cad8bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc3b185faa4de38d8264765cad8bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260927 00:57:05.608891 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:05.608933 21948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:05.609776 21962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:05.612854 21962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:05.620118 21962 catalog_manager.cc:1347] Generated new cluster ID: 735a165d4bdc446ca8f4ea2099847759
I20260927 00:57:05.620216 21962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:05.633484 21962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:05.634044 21962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:05.652607 21962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2afc3b185faa4de38d8264765cad8bf2: Generated new TSK 0
I20260927 00:57:05.653839 21962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:05.663995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21869
2026-09-27T00:57:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:05 dist-test-slave-9fh2 krb5kdc[21968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:05 dist-test-slave-9fh2 krb5kdc[21968](info): set up 2 sockets
Sep 27 00:57:05 dist-test-slave-9fh2 krb5kdc[21968](info): commencing operation
krb5kdc: starting...
W20260927 00:57:06.655529 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:06 dist-test-slave-9fh2 krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:06.788194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:06.919775 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:06.920172 21981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:06.920221 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:06.924576 21981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:06.924661 21981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:06.924679 21981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:06.924697 21981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:06.924711 21981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:06.930935 21981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:06.932446 21981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:06.933736 21981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:06.944407 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:06.944406 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:06.944368 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:06.944902 21981 server_base.cc:1034] running on GCE node
I20260927 00:57:06.945612 21981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:06.946903 21981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:06.948088 21981 hybrid_clock.cc:648] HybridClock initialized: now 1790470626948062 us; error 54 us; skew 500 ppm
Sep 27 00:57:06 dist-test-slave-9fh2 krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:06.951107 21981 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:06.952147 21981 webserver.cc:460] Webserver started at http://127.0.98.62:41439/ using document root <none> and password file <none>
I20260927 00:57:06.952744 21981 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:06.952808 21981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:06.953006 21981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:06.955183 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "23f11188583f49baaf8210651fda3150"
format_stamp: "Formatted at 2026-09-27 00:57:06 on dist-test-slave-9fh2"
server_key: "4e423b6f485c38c4fcb1ec5d80fa70d3"
server_key_iv: "44b1495d9629d7b61f814b17e671e3dc"
server_key_version: "encryptionkey@0"
I20260927 00:57:06.955763 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "23f11188583f49baaf8210651fda3150"
format_stamp: "Formatted at 2026-09-27 00:57:06 on dist-test-slave-9fh2"
server_key: "4e423b6f485c38c4fcb1ec5d80fa70d3"
server_key_iv: "44b1495d9629d7b61f814b17e671e3dc"
server_key_version: "encryptionkey@0"
I20260927 00:57:06.960071 21981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:57:06.963088 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:06.964812 21981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:57:06.964956 21981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "23f11188583f49baaf8210651fda3150"
format_stamp: "Formatted at 2026-09-27 00:57:06 on dist-test-slave-9fh2"
server_key: "4e423b6f485c38c4fcb1ec5d80fa70d3"
server_key_iv: "44b1495d9629d7b61f814b17e671e3dc"
server_key_version: "encryptionkey@0"
I20260927 00:57:06.965090 21981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:06.982674 21981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:06.986333 21981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:06.986503 21981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:06.993908 21981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36237
I20260927 00:57:06.993980 22048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36237 every 8 connection(s)
I20260927 00:57:06.995059 21981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:06.996498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 21981
I20260927 00:57:06.996671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:06.997087 392 external_mini_cluster.cc:1428] Setting key 64681145627612eed69bc677aad05af9
I20260927 00:57:06.999531 22049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 00:57:07 dist-test-slave-9fh2 krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:07.007503 22049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150: Bootstrap starting.
I20260927 00:57:07.011085 22049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:07.012218 22049 log.cc:824] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:07.013898 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:06.998879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49457 (local address 127.0.98.62:36237)
0927 00:57:06.999510 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:06.999528 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:06.999571 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:57:07.000310 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:07.000329 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:07.000735 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:07.001070 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:07.001922 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:07.002987 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:07.003889 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:07.004412 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:07.004565 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:07.007915 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:07.007922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:07.007940 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:07.007976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:07.010044 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:07.010919 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:07.010925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:07.010930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:07.011074 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:07.011574 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:07.011577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:07.011578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:07.012062 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:07.012248 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:07.012736 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 00:57:07.013029 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":240,"threads_started":1}
I20260927 00:57:07.015313 22049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150: No bootstrap required, opened a new log
I20260927 00:57:07.018141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:07.020787 22049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f11188583f49baaf8210651fda3150" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36237 } }
I20260927 00:57:07.021205 22049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:07.021343 22049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f11188583f49baaf8210651fda3150, State: Initialized, Role: FOLLOWER
I20260927 00:57:07.021950 22049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f11188583f49baaf8210651fda3150" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36237 } }
I20260927 00:57:07.022122 22049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:07.022193 22049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:07.022310 22049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:07.023752 22049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f11188583f49baaf8210651fda3150" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36237 } }
I20260927 00:57:07.024184 22049 leader_election.cc:302] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f11188583f49baaf8210651fda3150; no voters:
I20260927 00:57:07.024629 22049 leader_election.cc:288] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:07.024863 22059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:07.025275 22059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [term 1 LEADER]: Becoming Leader. State: Replica: 23f11188583f49baaf8210651fda3150, State: Running, Role: LEADER
I20260927 00:57:07.025830 22059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f11188583f49baaf8210651fda3150" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36237 } }
I20260927 00:57:07.026273 22049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:07.028337 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23f11188583f49baaf8210651fda3150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f11188583f49baaf8210651fda3150" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36237 } } }
I20260927 00:57:07.028525 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:07.029808 22061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f11188583f49baaf8210651fda3150. Latest consensus state: current_term: 1 leader_uuid: "23f11188583f49baaf8210651fda3150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f11188583f49baaf8210651fda3150" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36237 } } }
I20260927 00:57:07.029950 22061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:07.032099 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:07.021022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43027 (local address 127.0.98.62:36237)
0927 00:57:07.021203 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:07.021208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:07.021297 (+ 89us) server_negotiation.cc:407] Connection header received
0927 00:57:07.021448 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:07.021451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:07.021514 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:07.021635 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:07.022431 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:07.023244 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:07.024261 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:07.024627 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:07.024747 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:07.026655 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:07.026659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:07.026662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:07.026698 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:07.029542 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:07.030104 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:07.030108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:07.030109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:07.030208 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:07.030580 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:07.030583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:07.030585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:07.030829 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:07.030979 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:07.031790 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 00:57:07.031934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 00:57:07.032565 22074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:07.032658 22074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:07.032747 22073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:07.033651 22073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:07.040697 22073 catalog_manager.cc:1347] Generated new cluster ID: 6313cd1186c246a495e1c96bf68caf91
I20260927 00:57:07.040799 22073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:07.046281 22073 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:07.047513 22073 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:07.054075 22073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f11188583f49baaf8210651fda3150: Generated new TSK 0
I20260927 00:57:07.054965 22073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:07.064960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 21981
W20260927 00:57:07.071493 21966 connection.cc:535] client connection to 127.0.98.62:36237 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36237): Connection reset by peer (error 104)
2026-09-27T00:57:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:07 dist-test-slave-9fh2 krb5kdc[22080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:07 dist-test-slave-9fh2 krb5kdc[22080](info): set up 2 sockets
Sep 27 00:57:07 dist-test-slave-9fh2 krb5kdc[22080](info): commencing operation
krb5kdc: starting...
W20260927 00:57:08.039674 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:08 dist-test-slave-9fh2 krb5kdc[22080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:08.178366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:08.297269 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:08.297696 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:08.297777 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:08.302003 22093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:08.302088 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:08.302107 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:08.302124 22093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:08.302137 22093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:08.307749 22093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:08.308907 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:08.309984 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:08.319574 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:08.319590 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:08.319607 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:08.319852 22093 server_base.cc:1034] running on GCE node
I20260927 00:57:08.320601 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:08.322148 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:08.323360 22093 hybrid_clock.cc:648] HybridClock initialized: now 1790470628323330 us; error 49 us; skew 500 ppm
Sep 27 00:57:08 dist-test-slave-9fh2 krb5kdc[22080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:08.331116 22093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:08.332453 22093 webserver.cc:460] Webserver started at http://127.0.98.62:34907/ using document root <none> and password file <none>
I20260927 00:57:08.333173 22093 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:08.333257 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:08.333515 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:08.336218 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "cce5f1cb5a8846fab86a005045a5b84a"
format_stamp: "Formatted at 2026-09-27 00:57:08 on dist-test-slave-9fh2"
server_key: "7c1ff09ebac99b1cc113a581176194a7"
server_key_iv: "35679b0a787958c86fe993dfe2069e65"
server_key_version: "encryptionkey@0"
I20260927 00:57:08.336928 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "cce5f1cb5a8846fab86a005045a5b84a"
format_stamp: "Formatted at 2026-09-27 00:57:08 on dist-test-slave-9fh2"
server_key: "7c1ff09ebac99b1cc113a581176194a7"
server_key_iv: "35679b0a787958c86fe993dfe2069e65"
server_key_version: "encryptionkey@0"
I20260927 00:57:08.342057 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:57:08.346009 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:08.348097 22093 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:57:08.348306 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "cce5f1cb5a8846fab86a005045a5b84a"
format_stamp: "Formatted at 2026-09-27 00:57:08 on dist-test-slave-9fh2"
server_key: "7c1ff09ebac99b1cc113a581176194a7"
server_key_iv: "35679b0a787958c86fe993dfe2069e65"
server_key_version: "encryptionkey@0"
I20260927 00:57:08.348480 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:08.372737 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:08.376886 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:08.377143 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:08.385936 22093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36179
I20260927 00:57:08.385975 22160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36179 every 8 connection(s)
I20260927 00:57:08.387250 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:08.390462 22161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:08.396360 22161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a: Bootstrap starting.
I20260927 00:57:08.397159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22093
I20260927 00:57:08.397293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:08.397636 392 external_mini_cluster.cc:1428] Setting key 5635dab490e3b136eb398fab3d4bbe8d
I20260927 00:57:08.399420 22161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:08.400177 22161 log.cc:824] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:08.402345 22161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a: No bootstrap required, opened a new log
I20260927 00:57:08.405944 22161 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce5f1cb5a8846fab86a005045a5b84a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260927 00:57:08.406267 22161 raft_consensus.cc:374] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:08.406320 22161 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce5f1cb5a8846fab86a005045a5b84a, State: Initialized, Role: FOLLOWER
I20260927 00:57:08.406838 22161 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce5f1cb5a8846fab86a005045a5b84a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260927 00:57:08.406991 22161 raft_consensus.cc:388] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:08.407068 22161 raft_consensus.cc:482] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:08.407194 22161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:08.408262 22161 raft_consensus.cc:504] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce5f1cb5a8846fab86a005045a5b84a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260927 00:57:08.408605 22161 leader_election.cc:302] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cce5f1cb5a8846fab86a005045a5b84a; no voters:
I20260927 00:57:08.408934 22161 leader_election.cc:288] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:08.409034 22166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:08.409226 22166 raft_consensus.cc:686] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [term 1 LEADER]: Becoming Leader. State: Replica: cce5f1cb5a8846fab86a005045a5b84a, State: Running, Role: LEADER
I20260927 00:57:08.409479 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:08.399249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57853 (local address 127.0.98.62:36179)
0927 00:57:08.399824 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:08.399844 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:08.399898 (+ 54us) server_negotiation.cc:407] Connection header received
0927 00:57:08.400840 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:08.400865 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:08.401323 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:08.401703 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:08.402138 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:08.403319 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:08.404621 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:08.405742 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:08.405969 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:08.407005 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:08.407013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:08.407035 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:08.407640 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:08.407714 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:08.407741 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:08.407853 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:08.408073 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:57:08.408241 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":204,"threads_started":1}
I20260927 00:57:08.409608 22166 consensus_queue.cc:237] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce5f1cb5a8846fab86a005045a5b84a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260927 00:57:08.410336 22161 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:08.411425 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cce5f1cb5a8846fab86a005045a5b84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce5f1cb5a8846fab86a005045a5b84a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } } }
I20260927 00:57:08.411665 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:08.412149 22175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:08.413062 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cce5f1cb5a8846fab86a005045a5b84a. Latest consensus state: current_term: 1 leader_uuid: "cce5f1cb5a8846fab86a005045a5b84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce5f1cb5a8846fab86a005045a5b84a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } } }
I20260927 00:57:08.413208 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:08.413484 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:08.416249 22175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:08.423928 22175 catalog_manager.cc:1347] Generated new cluster ID: a18cf49481f245548f662d4714c40d1a
I20260927 00:57:08.424101 22175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:08.425886 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:08.416087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50257 (local address 127.0.98.62:36179)
0927 00:57:08.416216 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:08.416220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:08.420377 (+ 4157us) server_negotiation.cc:407] Connection header received
0927 00:57:08.420527 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:08.420531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:08.420612 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:08.420741 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:08.421105 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:08.422232 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:08.423642 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:08.424001 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:08.424102 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:08.425341 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:08.425346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:08.425349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:08.425368 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:08.425407 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:08.425421 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:08.425519 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:08.425672 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 00:57:08.425720 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:57:08.432749 22175 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:08.434136 22175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:08.444234 22175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cce5f1cb5a8846fab86a005045a5b84a: Generated new TSK 0
I20260927 00:57:08.445261 22175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:08.450234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22093
W20260927 00:57:08.455432 22078 connection.cc:535] client connection to 127.0.98.62:36179 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36179): Connection reset by peer (error 104)
2026-09-27T00:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:08 dist-test-slave-9fh2 krb5kdc[22192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:08 dist-test-slave-9fh2 krb5kdc[22192](info): set up 2 sockets
Sep 27 00:57:08 dist-test-slave-9fh2 krb5kdc[22192](info): commencing operation
krb5kdc: starting...
W20260927 00:57:09.419689 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:09 dist-test-slave-9fh2 krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:09.562721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39063
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:09.673866 22205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:09.674165 22205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:09.674227 22205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:09.678359 22205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:09.678431 22205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:09.678458 22205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:09.678479 22205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:09.678500 22205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:09.683969 22205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:09.685053 22205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:09.686055 22205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:09.695519 22210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:09.695534 22213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:09.695534 22211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:09.695749 22205 server_base.cc:1034] running on GCE node
I20260927 00:57:09.696516 22205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:09.697649 22205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:09.698863 22205 hybrid_clock.cc:648] HybridClock initialized: now 1790470629698835 us; error 47 us; skew 500 ppm
Sep 27 00:57:09 dist-test-slave-9fh2 krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:09.701753 22205 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:09.702702 22205 webserver.cc:460] Webserver started at http://127.0.98.62:44113/ using document root <none> and password file <none>
I20260927 00:57:09.703328 22205 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:09.703400 22205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:09.703594 22205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:09.705651 22205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "f0ce8df8b6d24b01931b1441b35bf139"
format_stamp: "Formatted at 2026-09-27 00:57:09 on dist-test-slave-9fh2"
server_key: "ed427df87c7a4d203b466c084af729e9"
server_key_iv: "745e54b1f9c07a1b996cc4e44f93f8b7"
server_key_version: "encryptionkey@0"
I20260927 00:57:09.706193 22205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0ce8df8b6d24b01931b1441b35bf139"
format_stamp: "Formatted at 2026-09-27 00:57:09 on dist-test-slave-9fh2"
server_key: "ed427df87c7a4d203b466c084af729e9"
server_key_iv: "745e54b1f9c07a1b996cc4e44f93f8b7"
server_key_version: "encryptionkey@0"
I20260927 00:57:09.710403 22205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 00:57:09.713268 22220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:09.714744 22205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 00:57:09.714913 22205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "f0ce8df8b6d24b01931b1441b35bf139"
format_stamp: "Formatted at 2026-09-27 00:57:09 on dist-test-slave-9fh2"
server_key: "ed427df87c7a4d203b466c084af729e9"
server_key_iv: "745e54b1f9c07a1b996cc4e44f93f8b7"
server_key_version: "encryptionkey@0"
I20260927 00:57:09.715027 22205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:09.719740 22205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:09.723007 22205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:09.723304 22205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:09.731403 22205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39063
I20260927 00:57:09.731763 22272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39063 every 8 connection(s)
I20260927 00:57:09.732462 22205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:09.736259 22273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:09.740658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22205
I20260927 00:57:09.740792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:09.741097 392 external_mini_cluster.cc:1428] Setting key c76857d25650670a116c462260dd03c3
I20260927 00:57:09.744402 22273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139: Bootstrap starting.
I20260927 00:57:09.747862 22273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:09.748721 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:09.742928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39353 (local address 127.0.98.62:39063)
0927 00:57:09.743555 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:09.743580 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:09.743636 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:57:09.744658 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:09.744684 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:09.745144 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:09.745613 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:09.746619 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:09.746626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:09.746673 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:09.746687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:09.746765 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:09.746796 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:09.746973 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:09.747227 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 00:57:09.747616 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":192,"threads_started":1}
I20260927 00:57:09.748943 22273 log.cc:824] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:09.752004 22273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139: No bootstrap required, opened a new log
I20260927 00:57:09.752741 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:09.757083 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:09.755201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35989 (local address 127.0.98.62:39063)
0927 00:57:09.755373 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:09.755378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:09.755607 (+ 229us) server_negotiation.cc:407] Connection header received
0927 00:57:09.755795 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:09.755798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:09.755859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:09.755936 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:09.756696 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:09.756699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:09.756702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:09.756713 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:09.756741 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:09.756753 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:09.756826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:09.756903 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 00:57:09.756945 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:57:09.757129 22273 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ce8df8b6d24b01931b1441b35bf139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260927 00:57:09.757524 22273 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:09.757656 22273 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ce8df8b6d24b01931b1441b35bf139, State: Initialized, Role: FOLLOWER
I20260927 00:57:09.758289 22273 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ce8df8b6d24b01931b1441b35bf139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260927 00:57:09.758494 22273 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:09.758571 22273 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:09.758693 22273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:09.760058 22273 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ce8df8b6d24b01931b1441b35bf139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260927 00:57:09.760488 22273 leader_election.cc:302] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0ce8df8b6d24b01931b1441b35bf139; no voters:
I20260927 00:57:09.760876 22273 leader_election.cc:288] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:09.760968 22283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:09.761234 22283 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [term 1 LEADER]: Becoming Leader. State: Replica: f0ce8df8b6d24b01931b1441b35bf139, State: Running, Role: LEADER
I20260927 00:57:09.761677 22283 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ce8df8b6d24b01931b1441b35bf139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } }
I20260927 00:57:09.762590 22273 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:09.763278 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0ce8df8b6d24b01931b1441b35bf139. Latest consensus state: current_term: 1 leader_uuid: "f0ce8df8b6d24b01931b1441b35bf139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ce8df8b6d24b01931b1441b35bf139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } } }
I20260927 00:57:09.763420 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:09.763700 22284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0ce8df8b6d24b01931b1441b35bf139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ce8df8b6d24b01931b1441b35bf139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39063 } } }
I20260927 00:57:09.763928 22284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:09.764058 22288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:09.768187 22288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:09.775512 22288 catalog_manager.cc:1347] Generated new cluster ID: 3dbe5fefdcac4aaaac21e4d4174b6186
I20260927 00:57:09.775616 22288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:09.810242 22288 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:09.810714 22288 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:09.815912 22288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0ce8df8b6d24b01931b1441b35bf139: Generated new TSK 0
I20260927 00:57:09.816879 22288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:09.830873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22205
2026-09-27T00:57:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:09 dist-test-slave-9fh2 krb5kdc[22304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:09 dist-test-slave-9fh2 krb5kdc[22304](info): set up 2 sockets
Sep 27 00:57:09 dist-test-slave-9fh2 krb5kdc[22304](info): commencing operation
krb5kdc: starting...
W20260927 00:57:10.807955 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:10 dist-test-slave-9fh2 krb5kdc[22304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:10.938757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:11.070863 22317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:11.071309 22317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:11.071376 22317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:11.075809 22317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:11.075913 22317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:11.075943 22317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:11.075966 22317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:11.075985 22317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:11.082283 22317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:11.083822 22317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:11.085217 22317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:11.095431 22325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:11.095466 22322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:11.095562 22323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:11.096081 22317 server_base.cc:1034] running on GCE node
I20260927 00:57:11.096840 22317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:11.098259 22317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:11.099468 22317 hybrid_clock.cc:648] HybridClock initialized: now 1790470631099433 us; error 50 us; skew 500 ppm
Sep 27 00:57:11 dist-test-slave-9fh2 krb5kdc[22304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:11.102761 22317 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:11.103879 22317 webserver.cc:460] Webserver started at http://127.0.98.62:39911/ using document root <none> and password file <none>
I20260927 00:57:11.104477 22317 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:11.104627 22317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:11.104862 22317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:11.106985 22317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "908748f6138343ae8459808aa4921daf"
format_stamp: "Formatted at 2026-09-27 00:57:11 on dist-test-slave-9fh2"
server_key: "b79b490158494b4d1e18d1cbf727089e"
server_key_iv: "ab425dfa98c125b51e9ed980449ad65b"
server_key_version: "encryptionkey@0"
I20260927 00:57:11.107553 22317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "908748f6138343ae8459808aa4921daf"
format_stamp: "Formatted at 2026-09-27 00:57:11 on dist-test-slave-9fh2"
server_key: "b79b490158494b4d1e18d1cbf727089e"
server_key_iv: "ab425dfa98c125b51e9ed980449ad65b"
server_key_version: "encryptionkey@0"
I20260927 00:57:11.111675 22317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:57:11.114418 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:11.115959 22317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:11.116101 22317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "908748f6138343ae8459808aa4921daf"
format_stamp: "Formatted at 2026-09-27 00:57:11 on dist-test-slave-9fh2"
server_key: "b79b490158494b4d1e18d1cbf727089e"
server_key_iv: "ab425dfa98c125b51e9ed980449ad65b"
server_key_version: "encryptionkey@0"
I20260927 00:57:11.116225 22317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:11.128322 22317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:11.131721 22317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:11.131892 22317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:11.139528 22317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41877
I20260927 00:57:11.139658 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41877 every 8 connection(s)
I20260927 00:57:11.140575 22317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:11.144392 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:11.147083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22317
I20260927 00:57:11.147261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:11.147568 392 external_mini_cluster.cc:1428] Setting key 9db1632b726361673432fbe1dd0d22b4
I20260927 00:57:11.151949 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf: Bootstrap starting.
I20260927 00:57:11.155185 22385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:11.156252 22385 log.cc:824] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:57:11 dist-test-slave-9fh2 krb5kdc[22304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:11.158964 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf: No bootstrap required, opened a new log
I20260927 00:57:11.163585 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:11.149097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56345 (local address 127.0.98.62:41877)
0927 00:57:11.149556 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:11.149573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:11.149612 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:57:11.150723 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:11.150744 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:11.151118 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:11.151493 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:11.152579 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:11.153674 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:11.154950 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:11.155589 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:11.155739 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:11.158696 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:11.158702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:11.158719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:11.158750 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:11.160606 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:11.161068 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:11.161073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:11.161077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:11.161206 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:11.161472 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:11.161476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:11.161477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:11.161921 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:11.162092 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:11.162399 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 00:57:11.162706 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260927 00:57:11.163784 22385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908748f6138343ae8459808aa4921daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260927 00:57:11.164158 22385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:11.164207 22385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908748f6138343ae8459808aa4921daf, State: Initialized, Role: FOLLOWER
I20260927 00:57:11.164743 22385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908748f6138343ae8459808aa4921daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260927 00:57:11.165005 22385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:11.165073 22385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:11.165181 22385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:11.166532 22385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908748f6138343ae8459808aa4921daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260927 00:57:11.167002 22385 leader_election.cc:302] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 908748f6138343ae8459808aa4921daf; no voters:
I20260927 00:57:11.167321 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:11.167471 22385 leader_election.cc:288] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:11.167685 22390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:11.168087 22390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [term 1 LEADER]: Becoming Leader. State: Replica: 908748f6138343ae8459808aa4921daf, State: Running, Role: LEADER
I20260927 00:57:11.168591 22390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908748f6138343ae8459808aa4921daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260927 00:57:11.169075 22385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:11.170589 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 908748f6138343ae8459808aa4921daf. Latest consensus state: current_term: 1 leader_uuid: "908748f6138343ae8459808aa4921daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908748f6138343ae8459808aa4921daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } } }
I20260927 00:57:11.170769 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:11.171186 22391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "908748f6138343ae8459808aa4921daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908748f6138343ae8459808aa4921daf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } } }
I20260927 00:57:11.171285 22391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:11.171466 22405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:11.174592 22405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:11.181346 22405 catalog_manager.cc:1347] Generated new cluster ID: ef204847b7474d1e8f10e8a362f21e70
I20260927 00:57:11.181416 22405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:11.185269 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:11.170456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35393 (local address 127.0.98.62:41877)
0927 00:57:11.170564 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:11.170568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:11.175520 (+ 4952us) server_negotiation.cc:407] Connection header received
0927 00:57:11.175663 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:11.175667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:11.175745 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:11.175895 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:11.176814 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:11.177717 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:11.178920 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:11.179363 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:11.179495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:11.181243 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:11.181248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:11.181252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:11.181283 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:11.183297 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:11.183842 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:11.183846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:11.183848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:11.183992 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:11.184263 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:11.184266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:11.184268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:11.184541 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:11.184691 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:11.184925 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:57:11.185078 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:57:11.193100 22405 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:11.194131 22405 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:11.202064 22405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 908748f6138343ae8459808aa4921daf: Generated new TSK 0
I20260927 00:57:11.202847 22405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:11.210822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22317
2026-09-27T00:57:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:11 dist-test-slave-9fh2 krb5kdc[22416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:11 dist-test-slave-9fh2 krb5kdc[22416](info): set up 2 sockets
Sep 27 00:57:11 dist-test-slave-9fh2 krb5kdc[22416](info): commencing operation
krb5kdc: starting...
W20260927 00:57:12.183785 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:12 dist-test-slave-9fh2 krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:12.327991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:12.463121 22429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:12.463524 22429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:12.463594 22429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:12.468083 22429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:12.468194 22429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:12.468225 22429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:12.468247 22429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:12.468266 22429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:12.474474 22429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:12.476054 22429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:12.477380 22429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:12.487597 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:12.487597 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:12.487861 22429 server_base.cc:1034] running on GCE node
W20260927 00:57:12.487591 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:12.488699 22429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:12.489996 22429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:12.491236 22429 hybrid_clock.cc:648] HybridClock initialized: now 1790470632491211 us; error 46 us; skew 500 ppm
Sep 27 00:57:12 dist-test-slave-9fh2 krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:12.494331 22429 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:12.495420 22429 webserver.cc:460] Webserver started at http://127.0.98.62:37421/ using document root <none> and password file <none>
I20260927 00:57:12.496021 22429 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:12.496089 22429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:12.496302 22429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:12.498507 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "d729c061077e4d379ed82fb4c799d7bd"
format_stamp: "Formatted at 2026-09-27 00:57:12 on dist-test-slave-9fh2"
server_key: "e9a01e26c405e0d3c0e42330fd3216a7"
server_key_iv: "48d29fbe26d1a2d98a0690f665f063fc"
server_key_version: "encryptionkey@0"
I20260927 00:57:12.499074 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "d729c061077e4d379ed82fb4c799d7bd"
format_stamp: "Formatted at 2026-09-27 00:57:12 on dist-test-slave-9fh2"
server_key: "e9a01e26c405e0d3c0e42330fd3216a7"
server_key_iv: "48d29fbe26d1a2d98a0690f665f063fc"
server_key_version: "encryptionkey@0"
I20260927 00:57:12.503253 22429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:57:12.506156 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:12.507723 22429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:12.507860 22429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "d729c061077e4d379ed82fb4c799d7bd"
format_stamp: "Formatted at 2026-09-27 00:57:12 on dist-test-slave-9fh2"
server_key: "e9a01e26c405e0d3c0e42330fd3216a7"
server_key_iv: "48d29fbe26d1a2d98a0690f665f063fc"
server_key_version: "encryptionkey@0"
I20260927 00:57:12.507994 22429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:12.526819 22429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:12.530164 22429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:12.530365 22429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:12.537818 22429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40683
I20260927 00:57:12.537950 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40683 every 8 connection(s)
I20260927 00:57:12.538890 22429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:12.542670 22497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:12.546280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22429
I20260927 00:57:12.546415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:12.546695 392 external_mini_cluster.cc:1428] Setting key c38a340cee2fcaf9eace091ad7183c8d
I20260927 00:57:12.550434 22497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd: Bootstrap starting.
I20260927 00:57:12.553730 22497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:12.554853 22497 log.cc:824] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:12.557765 22497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd: No bootstrap required, opened a new log
I20260927 00:57:12.558441 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:12.548410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45545 (local address 127.0.98.62:40683)
0927 00:57:12.549013 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:12.549031 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:12.549348 (+ 317us) server_negotiation.cc:407] Connection header received
0927 00:57:12.550495 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:12.550519 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:12.551041 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:12.551535 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:12.552137 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:12.553167 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:12.554480 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:12.555074 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:12.555252 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:12.556324 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:12.556330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:12.556346 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:12.556839 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:12.556906 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:12.556933 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:12.557017 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:12.557186 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 00:57:12.557400 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":187,"threads_started":1}
I20260927 00:57:12.562152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:12.562708 22497 raft_consensus.cc:348] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d729c061077e4d379ed82fb4c799d7bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40683 } }
I20260927 00:57:12.563074 22497 raft_consensus.cc:374] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:12.563126 22497 raft_consensus.cc:729] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d729c061077e4d379ed82fb4c799d7bd, State: Initialized, Role: FOLLOWER
I20260927 00:57:12.563746 22497 consensus_queue.cc:260] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d729c061077e4d379ed82fb4c799d7bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40683 } }
I20260927 00:57:12.563987 22497 raft_consensus.cc:388] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:12.564071 22497 raft_consensus.cc:482] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:12.564184 22497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:12.565662 22497 raft_consensus.cc:504] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d729c061077e4d379ed82fb4c799d7bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40683 } }
I20260927 00:57:12.566146 22497 leader_election.cc:302] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d729c061077e4d379ed82fb4c799d7bd; no voters:
I20260927 00:57:12.566588 22497 leader_election.cc:288] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:12.566745 22507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:12.567054 22507 raft_consensus.cc:686] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [term 1 LEADER]: Becoming Leader. State: Replica: d729c061077e4d379ed82fb4c799d7bd, State: Running, Role: LEADER
I20260927 00:57:12.567520 22507 consensus_queue.cc:237] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d729c061077e4d379ed82fb4c799d7bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40683 } }
I20260927 00:57:12.568238 22497 sys_catalog.cc:564] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:12.569710 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d729c061077e4d379ed82fb4c799d7bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d729c061077e4d379ed82fb4c799d7bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40683 } } }
I20260927 00:57:12.569885 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:12.570050 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:12.564951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60077 (local address 127.0.98.62:40683)
0927 00:57:12.565107 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:12.565113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:12.565242 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:57:12.565359 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:12.565362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:12.565427 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:12.565541 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:12.565934 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:12.566727 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:12.568009 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:12.568366 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:12.568483 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:12.569318 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:12.569323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:12.569326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:12.569342 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:12.569384 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:12.569400 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:12.569471 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:12.569826 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:57:12.569908 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:57:12.570684 22508 sys_catalog.cc:455] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d729c061077e4d379ed82fb4c799d7bd. Latest consensus state: current_term: 1 leader_uuid: "d729c061077e4d379ed82fb4c799d7bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d729c061077e4d379ed82fb4c799d7bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40683 } } }
I20260927 00:57:12.571702 22508 sys_catalog.cc:458] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd [sys.catalog]: This master's current role is: LEADER
W20260927 00:57:12.574641 22522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:12.574756 22522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:12.574884 22517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:12.575881 22517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:12.583695 22517 catalog_manager.cc:1347] Generated new cluster ID: 57b92228fc8544a4bb1ae2b99945da5f
I20260927 00:57:12.583817 22517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:12.596266 22517 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:12.597587 22517 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:12.607313 22517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d729c061077e4d379ed82fb4c799d7bd: Generated new TSK 0
I20260927 00:57:12.608482 22517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:12.617883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22429
2026-09-27T00:57:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:12 dist-test-slave-9fh2 krb5kdc[22528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:12 dist-test-slave-9fh2 krb5kdc[22528](info): set up 2 sockets
Sep 27 00:57:12 dist-test-slave-9fh2 krb5kdc[22528](info): commencing operation
krb5kdc: starting...
W20260927 00:57:13.583957 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:13 dist-test-slave-9fh2 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:13.724457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45843
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:13.843505 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:13.843819 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:13.843878 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:13.848104 22541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:13.848191 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:13.848217 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:13.848239 22541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:13.848258 22541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:13.853855 22541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:13.855017 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:13.856098 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:13.865171 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:13.865171 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:13.865164 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:13.865844 22541 server_base.cc:1034] running on GCE node
I20260927 00:57:13.866606 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:13.868125 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:13.869345 22541 hybrid_clock.cc:648] HybridClock initialized: now 1790470633869302 us; error 53 us; skew 500 ppm
Sep 27 00:57:13 dist-test-slave-9fh2 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:13.876494 22541 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:13.877790 22541 webserver.cc:460] Webserver started at http://127.0.98.62:35571/ using document root <none> and password file <none>
I20260927 00:57:13.878551 22541 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:13.878628 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:13.878866 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:13.881428 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "9c42e4cc4e124d29aa060dd5e3a089fd"
format_stamp: "Formatted at 2026-09-27 00:57:13 on dist-test-slave-9fh2"
server_key: "a9d460e3b351ee76c7fe3c87bdc49871"
server_key_iv: "0d889973c18d5994bcc794f3697804ce"
server_key_version: "encryptionkey@0"
I20260927 00:57:13.882102 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c42e4cc4e124d29aa060dd5e3a089fd"
format_stamp: "Formatted at 2026-09-27 00:57:13 on dist-test-slave-9fh2"
server_key: "a9d460e3b351ee76c7fe3c87bdc49871"
server_key_iv: "0d889973c18d5994bcc794f3697804ce"
server_key_version: "encryptionkey@0"
I20260927 00:57:13.887014 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 00:57:13.890400 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:13.892429 22541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:57:13.892629 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "9c42e4cc4e124d29aa060dd5e3a089fd"
format_stamp: "Formatted at 2026-09-27 00:57:13 on dist-test-slave-9fh2"
server_key: "a9d460e3b351ee76c7fe3c87bdc49871"
server_key_iv: "0d889973c18d5994bcc794f3697804ce"
server_key_version: "encryptionkey@0"
I20260927 00:57:13.892786 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:13.898615 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:13.902688 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:13.902912 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:13.911386 22541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45843
I20260927 00:57:13.911453 22608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45843 every 8 connection(s)
I20260927 00:57:13.912535 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:13.916229 22609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:13.922789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22541
I20260927 00:57:13.922923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:13.923260 392 external_mini_cluster.cc:1428] Setting key 83fe4ac9997bc45cedd416ad97eeb25b
I20260927 00:57:13.923439 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd: Bootstrap starting.
I20260927 00:57:13.926934 22609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:13.928069 22609 log.cc:824] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:13.930195 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:13.924962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55791 (local address 127.0.98.62:45843)
0927 00:57:13.925571 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:13.925600 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:13.925655 (+ 55us) server_negotiation.cc:407] Connection header received
0927 00:57:13.926667 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:13.926697 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:13.927239 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:13.927746 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:13.928739 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:13.928747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:13.928765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:13.928781 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:13.928873 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:13.928902 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:13.929022 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:13.929134 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 00:57:13.929350 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":164,"threads_started":1}
I20260927 00:57:13.930758 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd: No bootstrap required, opened a new log
I20260927 00:57:13.933970 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:13.935832 22609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45843 } }
I20260927 00:57:13.936174 22609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:57:13.936235 22609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c42e4cc4e124d29aa060dd5e3a089fd, State: Initialized, Role: FOLLOWER
I20260927 00:57:13.936859 22609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45843 } }
I20260927 00:57:13.937057 22609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:57:13.937136 22609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:57:13.937252 22609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:13.938382 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:13.936292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39397 (local address 127.0.98.62:45843)
0927 00:57:13.936430 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:13.936434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:13.936734 (+ 300us) server_negotiation.cc:407] Connection header received
0927 00:57:13.936870 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:13.936873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:13.936915 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:13.937011 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:13.937959 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:13.937963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:13.937966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:57:13.937979 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:13.938017 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:57:13.938028 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:13.938110 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:13.938195 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:57:13.938241 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:57:13.938619 22609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45843 } }
I20260927 00:57:13.939086 22609 leader_election.cc:302] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c42e4cc4e124d29aa060dd5e3a089fd; no voters:
I20260927 00:57:13.939569 22609 leader_election.cc:288] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:57:13.939709 22619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:13.939988 22619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [term 1 LEADER]: Becoming Leader. State: Replica: 9c42e4cc4e124d29aa060dd5e3a089fd, State: Running, Role: LEADER
I20260927 00:57:13.940397 22619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45843 } }
I20260927 00:57:13.941115 22609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:13.942163 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45843 } } }
I20260927 00:57:13.942343 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:13.942651 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c42e4cc4e124d29aa060dd5e3a089fd. Latest consensus state: current_term: 1 leader_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c42e4cc4e124d29aa060dd5e3a089fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45843 } } }
I20260927 00:57:13.942751 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:13.942921 22628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:13.946973 22628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:13.954885 22628 catalog_manager.cc:1347] Generated new cluster ID: 2ee64eb8af8948c4a2c88cdb59d2c703
I20260927 00:57:13.954994 22628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:13.972388 22628 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:13.972831 22628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:13.980038 22628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c42e4cc4e124d29aa060dd5e3a089fd: Generated new TSK 0
I20260927 00:57:13.981027 22628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:57:13.988997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22541
2026-09-27T00:57:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1370 ms)
[----------] 18 tests from AuthTokenIssuingTest (24908 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:14 dist-test-slave-9fh2 krb5kdc[22640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:14 dist-test-slave-9fh2 krb5kdc[22640](info): set up 2 sockets
Sep 27 00:57:14 dist-test-slave-9fh2 krb5kdc[22640](info): commencing operation
krb5kdc: starting...
W20260927 00:57:14.955667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:15.091745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34947
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34947,127.0.98.61:41657,127.0.98.60:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:15.222345 22653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:57:15.222771 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:15.222798 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:15.222851 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:15.227528 22653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:15.227639 22653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:15.227660 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:15.227676 22653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:15.227694 22653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:15.234033 22653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34947,127.0.98.61:41657,127.0.98.60:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:15.235642 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:15.237035 22653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:15.247859 22658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:15.247951 22661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:15.247882 22659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:15.248306 22653 server_base.cc:1034] running on GCE node
I20260927 00:57:15.248996 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:15.250238 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:15.251461 22653 hybrid_clock.cc:648] HybridClock initialized: now 1790470635251454 us; error 79 us; skew 500 ppm
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:15.255038 22653 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:15.256415 22653 webserver.cc:460] Webserver started at http://127.0.98.62:34203/ using document root <none> and password file <none>
I20260927 00:57:15.257094 22653 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:15.257221 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:15.257478 22653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:15.259974 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "aa04cd963d374ddc9a47c6fa439dba45"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "9eaa7cce83ee134616d511c7f823d407"
server_key_iv: "f3dac240c210926a804863f30c2b5858"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.260615 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa04cd963d374ddc9a47c6fa439dba45"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "9eaa7cce83ee134616d511c7f823d407"
server_key_iv: "f3dac240c210926a804863f30c2b5858"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.265513 22653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:57:15.268749 22668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:15.270486 22653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:57:15.270665 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "aa04cd963d374ddc9a47c6fa439dba45"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "9eaa7cce83ee134616d511c7f823d407"
server_key_iv: "f3dac240c210926a804863f30c2b5858"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.270833 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:15.290977 22653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:15.295025 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:15.295315 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:15.302865 22653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34947
I20260927 00:57:15.303252 22720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34947 every 8 connection(s)
I20260927 00:57:15.303931 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:15.307717 22721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:15.309756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22653
I20260927 00:57:15.309921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:15.310249 392 external_mini_cluster.cc:1428] Setting key b48056e4a9c4396c3cff3bedd209fe2d
I20260927 00:57:15.313491 22721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:15.333734 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.316858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.98.62:34947)
0927 00:57:15.317474 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.317498 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.317899 (+ 401us) server_negotiation.cc:407] Connection header received
0927 00:57:15.319350 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.319355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.319594 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.319795 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.320892 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.322014 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.323553 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.324153 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.324413 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.327567 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.327576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.327594 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.327630 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.329760 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.330473 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.330481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.330483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.330655 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.331059 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.331063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.331064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.331715 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.331847 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.332390 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.332671 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":184,"threads_started":1}
I20260927 00:57:15.333741 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.316859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34947 (local address 127.0.0.1:35436)
0927 00:57:15.317700 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.317751 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.318860 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.319819 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.319825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.320401 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.320724 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.320741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.322147 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.322150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.323325 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.323337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.324429 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.324433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.324560 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.325212 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.325231 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.327318 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.329952 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.329964 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.329977 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.330310 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.330776 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.330779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.330781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.330947 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.331868 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.331875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.332089 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.332288 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.332800 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":52,"spinlock_wait_cycles":66432,"thread_start_us":248,"threads_started":1}
I20260927 00:57:15.336845 22721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.337839 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.337488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48576)
0927 00:57:15.337668 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.337756 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:57:15.338068 22669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
I20260927 00:57:15.339277 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.338937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48590)
0927 00:57:15.339115 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.339201 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 00:57:15.339448 22721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:57:15.343693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41657
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34947,127.0.98.61:41657,127.0.98.60:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:57:15.388789 22721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } attempt: 1
I20260927 00:57:15.390054 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.389581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48600)
0927 00:57:15.389788 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.389949 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 00:57:15.391022 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.390733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48606)
0927 00:57:15.390847 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.390964 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:57:15.391181 22721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
I20260927 00:57:15.459353 22721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } attempt: 2
I20260927 00:57:15.460427 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.460052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48612)
0927 00:57:15.460221 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.460317 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:57:15.461138 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.460926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48614)
0927 00:57:15.461028 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.461078 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:57:15.461253 22721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41657: connect: Connection refused (error 111)
W20260927 00:57:15.461505 22728 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:57:15.461828 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:15.461872 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:15.461930 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:15.466032 22728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:15.466105 22728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:15.466131 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:15.466151 22728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:15.466172 22728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:15.471624 22728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34947,127.0.98.61:41657,127.0.98.60:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:15.472746 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:15.473775 22728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:57:15.483080 22728 server_base.cc:1034] running on GCE node
W20260927 00:57:15.483049 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:15.483186 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:15.483114 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:15.484000 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:15.485342 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:15.486580 22728 hybrid_clock.cc:648] HybridClock initialized: now 1790470635486554 us; error 51 us; skew 500 ppm
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:15.489645 22728 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:57:15.490581 22728 webserver.cc:460] Webserver started at http://127.0.98.61:41791/ using document root <none> and password file <none>
I20260927 00:57:15.491127 22728 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:15.491236 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:15.491443 22728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:15.493499 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data/instance:
uuid: "74c0128e97c044fc8a5cf89436e3bc66"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "d125c82a1a185dea533c340c947abad3"
server_key_iv: "5047ea0eccad390c38dd3e9178d95c92"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.494043 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal/instance:
uuid: "74c0128e97c044fc8a5cf89436e3bc66"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "d125c82a1a185dea533c340c947abad3"
server_key_iv: "5047ea0eccad390c38dd3e9178d95c92"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.498170 22728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:57:15.501202 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:15.502767 22728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:57:15.502918 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal
uuid: "74c0128e97c044fc8a5cf89436e3bc66"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "d125c82a1a185dea533c340c947abad3"
server_key_iv: "5047ea0eccad390c38dd3e9178d95c92"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.503038 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:15.525470 22728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:15.528749 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:15.528947 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:15.536223 22728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41657
I20260927 00:57:15.536589 22795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41657 every 8 connection(s)
I20260927 00:57:15.537204 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:57:15.540900 22796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:15.541991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22728
I20260927 00:57:15.542124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-1/wal/instance
I20260927 00:57:15.542407 392 external_mini_cluster.cc:1428] Setting key fb0fe200303277c079161e26be5090f9
I20260927 00:57:15.546394 22796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:57:15.552453 22721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:15.565372 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.549598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.98.62:34947)
0927 00:57:15.551292 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.551296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.551313 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:57:15.552733 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.552736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.552790 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.552835 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.554622 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.555397 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.557148 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.557393 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.557474 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.561326 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.561330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.561333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.561368 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.562831 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.563560 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.563565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.563567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.563648 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.563999 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.564001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.564003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.564219 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.564312 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.565001 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.565169 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1579}
I20260927 00:57:15.565551 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.553183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48630)
0927 00:57:15.553657 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.553674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.553779 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.554516 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.554519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.554752 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.554871 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.554878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.556451 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.556455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.557021 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.557031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.558313 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.558317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.558429 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.558809 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.558824 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.560416 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.563564 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.563568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.563571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.563878 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.564262 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.564265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.564267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.564353 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.564981 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.564985 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.565101 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.565206 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.565370 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"spinlock_wait_cycles":3456}
I20260927 00:57:15.566414 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.553613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.0.98.61:41657)
0927 00:57:15.554043 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.554054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.554082 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:57:15.554124 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.554129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.554311 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.554495 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.555043 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.556288 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.557614 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.558169 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.558282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.560645 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.560652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.560669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.560709 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.563413 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.564029 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.564036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.564037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.564149 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.564456 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.564459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.564461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.564840 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.564983 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.565305 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.565468 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":209,"threads_started":1}
I20260927 00:57:15.566839 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.550084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34947 (local address 127.0.0.1:35442)
0927 00:57:15.551053 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.551128 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.552330 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.553255 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.553264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.553783 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.554494 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.554512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.555522 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.555526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.557005 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.557022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.557574 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.557592 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.557860 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.558759 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.559426 (+ 667us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.561153 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.562950 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.562960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.562974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.563408 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.563733 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.563736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.563738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.563891 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.564370 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.564379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.564717 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.564907 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.565318 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":189,"thread_start_us":161,"threads_started":1}
I20260927 00:57:15.568831 22796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.568915 22721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.569751 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.569432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46822)
0927 00:57:15.569562 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.569625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:57:15.570518 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.570258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46834)
0927 00:57:15.570407 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.570458 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 00:57:15.570700 22721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:57:15.575202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40541
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34947,127.0.98.61:41657,127.0.98.60:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:57:15.578081 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.569499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.0.98.61:41657)
0927 00:57:15.569604 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.569608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.570090 (+ 482us) server_negotiation.cc:407] Connection header received
0927 00:57:15.570117 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.570119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.570158 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.570222 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.570655 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.571252 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.572165 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.572419 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.572501 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.574981 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.574987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.574990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.575021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.576471 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.576877 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.576881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.576883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.576962 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.577209 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.577212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.577214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.577418 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.577509 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.577764 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.577942 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":48}
I20260927 00:57:15.578081 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.569432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48638)
0927 00:57:15.569910 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.569925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.570015 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.570249 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.570252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.570434 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.570542 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.570549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.571383 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.571387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.572040 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.572054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.572572 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.572576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.572703 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.573285 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.573298 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.574721 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.576592 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.576595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.576598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.576784 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.577044 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.577047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.577049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.577131 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.577507 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.577510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.577612 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.577704 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.577877 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416}
I20260927 00:57:15.579376 22796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.580307 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.579947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46838)
0927 00:57:15.580123 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.580231 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 00:57:15.580565 22744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260927 00:57:15.581676 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.581426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46842)
0927 00:57:15.581561 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.581617 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 00:57:15.581841 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260927 00:57:15.601890 22721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 1
I20260927 00:57:15.603058 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.602656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46846)
0927 00:57:15.602867 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.602960 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 00:57:15.603938 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.603705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46850)
0927 00:57:15.603821 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.603879 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:57:15.604115 22721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260927 00:57:15.631259 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 1
I20260927 00:57:15.632421 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.632003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46858)
0927 00:57:15.632223 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.632312 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 00:57:15.633821 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.633538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46864)
0927 00:57:15.633695 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.633753 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 00:57:15.633980 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260927 00:57:15.679275 22721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 2
I20260927 00:57:15.680300 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.679844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46870)
0927 00:57:15.680084 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.680187 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 00:57:15.681486 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.681194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46880)
0927 00:57:15.681347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.681415 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:57:15.681649 22721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
W20260927 00:57:15.691628 22802 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:57:15.691937 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:15.691960 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:15.691998 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:15.696132 22802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:15.696208 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:15.696228 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:15.696244 22802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:15.696259 22802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:15.702118 22802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34947,127.0.98.61:41657,127.0.98.60:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:15.702234 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 2
I20260927 00:57:15.703485 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.702996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46892)
0927 00:57:15.703240 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.703327 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:57:15.703555 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:15.704576 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.704249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46896)
0927 00:57:15.704460 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.704514 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260927 00:57:15.704746 22796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40541: connect: Connection refused (error 111)
I20260927 00:57:15.704798 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:15.714453 22808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:15.714581 22802 server_base.cc:1034] running on GCE node
W20260927 00:57:15.714468 22809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:15.714468 22811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:15.715646 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:15.717221 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:15.718438 22802 hybrid_clock.cc:648] HybridClock initialized: now 1790470635718367 us; error 84 us; skew 500 ppm
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:15.722348 22802 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:57:15.723680 22802 webserver.cc:460] Webserver started at http://127.0.98.60:39221/ using document root <none> and password file <none>
I20260927 00:57:15.724383 22802 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:15.724467 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:15.724763 22802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:15.727417 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data/instance:
uuid: "7c87105e9b8d4a58b48a485b17315e47"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "e2476bc394dc08ff30a65fb601ed7bfa"
server_key_iv: "134e16e91f4478df04c627e7eb799373"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.728093 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal/instance:
uuid: "7c87105e9b8d4a58b48a485b17315e47"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "e2476bc394dc08ff30a65fb601ed7bfa"
server_key_iv: "134e16e91f4478df04c627e7eb799373"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.733233 22802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:57:15.736765 22818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:15.738571 22802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:57:15.738763 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal
uuid: "7c87105e9b8d4a58b48a485b17315e47"
format_stamp: "Formatted at 2026-09-27 00:57:15 on dist-test-slave-9fh2"
server_key: "e2476bc394dc08ff30a65fb601ed7bfa"
server_key_iv: "134e16e91f4478df04c627e7eb799373"
server_key_version: "encryptionkey@0"
I20260927 00:57:15.738921 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:15.754329 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:15.758481 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:15.758736 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:15.767786 22802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40541
I20260927 00:57:15.768131 22870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40541 every 8 connection(s)
I20260927 00:57:15.769057 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:57:15.772429 22871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:15.772505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 22802
I20260927 00:57:15.772616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470348013064-392-0/minicluster-data/master-2/wal/instance
I20260927 00:57:15.772907 392 external_mini_cluster.cc:1428] Setting key c86d41e9bef622d51a8c759c2bc751d0
I20260927 00:57:15.777163 22871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.780799 22721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 3
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:15.792054 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.781615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46916)
0927 00:57:15.781828 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.781843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.781931 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.782489 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.782493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.782699 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.782800 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.782807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.784132 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.784136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.784901 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.784910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.785548 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.785552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.785658 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.786033 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.786047 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.787984 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.790420 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.790424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.790426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.790699 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.791001 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.791004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.791006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.791125 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.791555 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.791559 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.791646 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.791731 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.791876 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 00:57:15.792328 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.779372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.0.98.62:34947)
0927 00:57:15.779533 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.779539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.780415 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:57:15.780682 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.780686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.780757 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.780841 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.782115 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.782763 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.784179 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.784530 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.784621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.788305 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.788308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.788311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.788336 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.789937 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.790557 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.790561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.790563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.790659 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.790981 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.790985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.790986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.791342 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.791451 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.791883 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.792182 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:57:15.792387 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.774808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.0.98.60:40541)
0927 00:57:15.775549 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.775570 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.775620 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:57:15.776436 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.776460 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.776878 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.777245 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.778225 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.779698 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.780999 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.782418 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.782935 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.785022 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.785029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.785045 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.785076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.787193 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.787603 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.787616 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.787621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.787759 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.788170 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.788175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.788178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.789794 (+ 1616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.790038 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.790458 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.790864 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":521,"thread_start_us":171,"threads_started":1}
I20260927 00:57:15.792369 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.781723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.0.98.60:40541)
0927 00:57:15.782231 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.782240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.782253 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:57:15.782291 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.782294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.782344 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.782442 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.782941 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.783975 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.785042 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.785416 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.785510 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.788163 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.788167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.788169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.788189 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.790286 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.790810 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.790812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.790814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.790887 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.791246 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.791249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.791251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.791468 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.791556 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.791794 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.791915 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":403,"spinlock_wait_cycles":1280,"thread_start_us":191,"threads_started":1}
I20260927 00:57:15.793327 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.779295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34947 (local address 127.0.0.1:35456)
0927 00:57:15.780224 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.780312 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.780543 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.781089 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.781096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.781704 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.781929 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.781940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.782942 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.782945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.784052 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.784062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.784901 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.784906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.785093 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.785838 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.785855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.788156 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.790061 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.790069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.790076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.790441 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.790765 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.790767 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.790769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.790857 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.791471 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.791478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.791633 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.792893 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.793138 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":24,"spinlock_wait_cycles":1408,"thread_start_us":144,"threads_started":1}
I20260927 00:57:15.795070 22871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.798985 22721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45: Bootstrap starting.
I20260927 00:57:15.799221 22796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } attempt: 3
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:57:15.802726 22721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:15.804019 22721 log.cc:824] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:57:15.806730 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.795653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41657 (local address 127.0.0.1:48650)
0927 00:57:15.796209 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.796223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.796347 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.796702 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.796705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.796895 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.797080 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.797088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.798817 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.798821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.799670 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.799681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.800430 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.800433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.800531 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.800952 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.800967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.802713 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.804910 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.804913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.804916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.805168 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.805492 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.805495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.805497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.805587 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.806165 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.806168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.806273 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.806377 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.806534 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484}
I20260927 00:57:15.807303 22721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45: No bootstrap required, opened a new log
I20260927 00:57:15.807446 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.795972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.0.98.61:41657)
0927 00:57:15.796117 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.796122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.796272 (+ 150us) server_negotiation.cc:407] Connection header received
0927 00:57:15.796450 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.796453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.796505 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.796556 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.797325 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.798103 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.799823 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.800184 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.800260 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.802938 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.802942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.802945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.802972 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.804550 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.805291 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.805295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.805297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.805384 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.805698 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.805701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.805703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.805963 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.806051 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.807120 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.807281 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:57:15.808404 22871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:15.809110 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.795637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.0.98.62:34947)
0927 00:57:15.795812 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.795816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.795874 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:57:15.796015 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.796018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.796066 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.796128 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.796734 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.797307 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.798691 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.799016 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.799287 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.802945 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.802949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.802951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.802976 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.806388 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.807161 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.807166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.807168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.807260 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.807821 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.807824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.808053 (+ 229us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.808285 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.808440 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.808832 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.808960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:57:15.809952 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.799715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46924)
0927 00:57:15.800013 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.800028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.800109 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.800671 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.800675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.800918 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.801043 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.801050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.802151 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.802155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.803113 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.803124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.804252 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.804256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.804351 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.804832 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.804843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.806450 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.808294 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.808297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.808298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.808512 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.808827 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.808830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.808832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.808918 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.809468 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.809471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.809569 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.809644 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.809790 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260927 00:57:15.810577 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.800264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.0.98.60:40541)
0927 00:57:15.800389 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.800393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.800406 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:57:15.800443 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.800446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.800492 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.801161 (+ 669us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.801254 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.802023 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.803388 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.804050 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.804629 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.806588 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.806592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.806595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.806617 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.808187 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.808641 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.808644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.808646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.808729 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.809022 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.809025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.809027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.809352 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.810226 (+ 874us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.810295 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.810428 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:57:15.813519 22721 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.813798 22721 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa04cd963d374ddc9a47c6fa439dba45, State: Initialized, Role: FOLLOWER
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:57:15.814152 22721 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.814679 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: Bootstrap starting.
I20260927 00:57:15.815768 22721 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:15.818004 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.808999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40541 (local address 127.0.0.1:46926)
0927 00:57:15.809501 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:15.809516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:15.809591 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:15.809908 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:15.809910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:15.810071 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:15.810183 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.810195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.811132 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.811135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.811899 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:15.811910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.812505 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.812510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:15.812578 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.813047 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:15.813060 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:15.814591 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:15.816552 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.816555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.816557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.816773 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.817021 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:15.817023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:15.817025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:15.817091 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:15.817482 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:15.817485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:15.817560 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:57:15.817667 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:57:15.817814 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424}
I20260927 00:57:15.818329 22796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:15.818646 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.809218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.0.98.60:40541)
0927 00:57:15.809346 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.809351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.809663 (+ 312us) server_negotiation.cc:407] Connection header received
0927 00:57:15.809689 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.809691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.809731 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.809789 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.810291 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.811023 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.812026 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.812318 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.812404 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.814722 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.814725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.814728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.814749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.816408 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.816865 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.816868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.816870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.816948 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.817168 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.817171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.817173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.817369 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.817453 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.818368 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.818484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:57:15.818691 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.818831 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.819514 22796 log.cc:824] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:57:15 dist-test-slave-9fh2 krb5kdc[22640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:57:15.822007 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: No bootstrap required, opened a new log
I20260927 00:57:15.822012 22871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: Bootstrap starting.
I20260927 00:57:15.825023 22871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:15.825546 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.811364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.98.61:41657)
0927 00:57:15.811677 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.811681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.811827 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:57:15.811997 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.812000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.812054 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.812104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.813297 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.814212 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.816012 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.816315 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.816397 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.821699 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.821703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.821706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.821739 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.823448 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.823967 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.823971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.823973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.824058 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.824365 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.824368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.824370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.824604 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.824785 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.825250 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.825385 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260927 00:57:15.825964 22871 log.cc:824] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:15.822021 22653 master_runner.cc:186] Error getting master registration for 127.0.98.61:41657: OK, instance_id { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" instance_seqno: 1790470635534163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:57:15.826577 22653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:57:15.826767 22796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.827206 22796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c0128e97c044fc8a5cf89436e3bc66, State: Initialized, Role: FOLLOWER
I20260927 00:57:15.827644 22796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.828318 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.828552 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.829423 22796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:57:15.830067 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:15.830159 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:15.830433 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:15.832155 22871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: No bootstrap required, opened a new log
I20260927 00:57:15.832144 22695 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aa04cd963d374ddc9a47c6fa439dba45"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35462
I20260927 00:57:15.832412 22695 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:57:15.832517 22695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:15.833940 22695 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.835001 22695 leader_election.cc:288] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [CANDIDATE]: Term 1 election: Requested vote from peers 74c0128e97c044fc8a5cf89436e3bc66 (127.0.98.61:41657), 7c87105e9b8d4a58b48a485b17315e47 (127.0.98.60:40541)
I20260927 00:57:15.836163 22770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa04cd963d374ddc9a47c6fa439dba45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74c0128e97c044fc8a5cf89436e3bc66"
I20260927 00:57:15.836433 22770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:15.836311 22871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.836545 22871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c87105e9b8d4a58b48a485b17315e47, State: Initialized, Role: FOLLOWER
I20260927 00:57:15.836874 22871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.837637 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.837920 22770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa04cd963d374ddc9a47c6fa439dba45 in term 1.
I20260927 00:57:15.837975 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.838418 22871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:15.838682 22669 leader_election.cc:302] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c0128e97c044fc8a5cf89436e3bc66, aa04cd963d374ddc9a47c6fa439dba45; no voters:
I20260927 00:57:15.838733 22845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa04cd963d374ddc9a47c6fa439dba45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c87105e9b8d4a58b48a485b17315e47"
I20260927 00:57:15.839011 22878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:15.839030 22845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:15.839287 22878 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [term 1 LEADER]: Becoming Leader. State: Replica: aa04cd963d374ddc9a47c6fa439dba45, State: Running, Role: LEADER
I20260927 00:57:15.839787 22878 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } }
I20260927 00:57:15.840812 22845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa04cd963d374ddc9a47c6fa439dba45 in term 1.
I20260927 00:57:15.842073 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa04cd963d374ddc9a47c6fa439dba45. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.842211 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:15.845504 22923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:15.846762 22923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 00:57:15.846774 22921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:15.846872 22921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:15.847937 22728 master_runner.cc:263] No leader master found from master 74c0128e97c044fc8a5cf89436e3bc66
I20260927 00:57:15.848021 22728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:57:15.856026 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:15.856138 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:15.857357 22845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [term 1 FOLLOWER]: Refusing update from remote peer aa04cd963d374ddc9a47c6fa439dba45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:57:15.858495 22770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [term 1 FOLLOWER]: Refusing update from remote peer aa04cd963d374ddc9a47c6fa439dba45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:57:15.859349 22906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:57:15.859869 22906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:57:15.864406 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa04cd963d374ddc9a47c6fa439dba45. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.864553 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.866199 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.852200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.0.98.61:41657)
0927 00:57:15.852318 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.852322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.856867 (+ 4545us) server_negotiation.cc:407] Connection header received
0927 00:57:15.857298 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.857301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.857353 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.857409 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.858569 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.859321 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.860455 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.860740 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.860819 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.862713 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.862717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.862720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.862751 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.864377 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.864856 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.864860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.864862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.864943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.865242 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.865245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.865246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.865468 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.865568 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.865934 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.866053 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:57:15.869445 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa04cd963d374ddc9a47c6fa439dba45. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.869652 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.871846 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.871999 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:15.872953 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.873144 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.873324 22928 mvcc.cc:205] Tried to move back new op lower bound from 7333767724463984640 to 7333767724404977664. Current Snapshot: MvccSnapshot[applied={T|T < 7333767724463984640}]
I20260927 00:57:15.875982 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.851700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.0.98.62:34947)
0927 00:57:15.851861 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.851866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.867126 (+ 15260us) server_negotiation.cc:407] Connection header received
0927 00:57:15.867372 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.867376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.867433 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.867484 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.868459 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.869128 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.870173 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.870500 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.870568 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.872425 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.872429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.872431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.872457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.874132 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.874619 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.874623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.874625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.874707 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.874995 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.874999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.875001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.875246 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.875343 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.875706 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.875827 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:57:15.876411 22932 mvcc.cc:205] Tried to move back new op lower bound from 7333767724463984640 to 7333767724404977664. Current Snapshot: MvccSnapshot[applied={T|T < 7333767724463984640}]
I20260927 00:57:15.878038 22923 catalog_manager.cc:1347] Generated new cluster ID: 6c8e756ae9ff41c496ee1d9f5d8efeb9
I20260927 00:57:15.878103 22923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:15.885084 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:15.852493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.0.98.60:40541)
0927 00:57:15.852675 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.852680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.857233 (+ 4553us) server_negotiation.cc:407] Connection header received
0927 00:57:15.858568 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.858572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.858655 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.858788 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.866102 (+ 7314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.867021 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.871278 (+ 4257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:15.871669 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:15.871763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:15.877011 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:15.877016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:15.877019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:15.877059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:15.878963 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.883321 (+ 4358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.883326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.883329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.883495 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:15.883877 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:15.883881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:15.883882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:15.884159 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:15.884300 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:15.884703 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 00:57:15.884881 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:57:15.894260 22923 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:15.895915 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.896072 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:15.897629 22930 mvcc.cc:205] Tried to move back new op lower bound from 7333767724463984640 to 7333767724404977664. Current Snapshot: MvccSnapshot[applied={T|T < 7333767724463984640}]
I20260927 00:57:15.898608 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa04cd963d374ddc9a47c6fa439dba45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa04cd963d374ddc9a47c6fa439dba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34947 } } peers { permanent_uuid: "74c0128e97c044fc8a5cf89436e3bc66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41657 } } peers { permanent_uuid: "7c87105e9b8d4a58b48a485b17315e47" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40541 } } }
I20260927 00:57:15.898753 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:15.899703 22923 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:15.921576 22923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa04cd963d374ddc9a47c6fa439dba45: Generated new TSK 0
I20260927 00:57:15.922552 22923 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:57:15.963837 22724 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.959961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.0.98.62:34947)
0927 00:57:15.960139 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.960145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.962456 (+ 2311us) server_negotiation.cc:407] Connection header received
0927 00:57:15.962716 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.962721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.962783 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.962844 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.963650 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35478: BlockingRecv error: recv got EOF from 127.0.0.1:35478 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 00:57:15.965870 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.961054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.0.98.61:41657)
0927 00:57:15.961190 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.961195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.964675 (+ 3480us) server_negotiation.cc:407] Connection header received
0927 00:57:15.964884 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.964887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.964953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.965028 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.965715 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48668: BlockingRecv error: recv got EOF from 127.0.0.1:48668 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:57:15.967839 22876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.961782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.98.60:40541)
0927 00:57:15.961936 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.961940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.966586 (+ 4646us) server_negotiation.cc:407] Connection header received
0927 00:57:15.966904 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.966908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.966972 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.967029 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.967682 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46948: BlockingRecv error: recv got EOF from 127.0.0.1:46948 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 00:57:15.974068 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.971917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.98.61:41657)
0927 00:57:15.972081 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.972087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.972730 (+ 643us) server_negotiation.cc:407] Connection header received
0927 00:57:15.973044 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.973047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.973156 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.973215 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.973916 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:57:15.980980 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.978984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.0.98.61:41657)
0927 00:57:15.979122 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.979127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.979792 (+ 665us) server_negotiation.cc:407] Connection header received
0927 00:57:15.980026 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.980030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.980110 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.980176 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.980806 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48690: BlockingRecv error: recv got EOF from 127.0.0.1:48690 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 00:57:15.988052 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.986041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.0.98.61:41657)
0927 00:57:15.986172 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.986178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.986817 (+ 639us) server_negotiation.cc:407] Connection header received
0927 00:57:15.987057 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.987062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.987126 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.987221 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.987891 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48702: BlockingRecv error: recv got EOF from 127.0.0.1:48702 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 00:57:15.997980 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:15.995717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.0.98.61:41657)
0927 00:57:15.995937 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:15.995942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:15.996669 (+ 727us) server_negotiation.cc:407] Connection header received
0927 00:57:15.996945 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:15.996949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:15.997023 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:15.997088 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:15.997821 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48704: BlockingRecv error: recv got EOF from 127.0.0.1:48704 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260927 00:57:16.012190 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:16.009955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.0.98.61:41657)
0927 00:57:16.010124 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:16.010129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:16.010871 (+ 742us) server_negotiation.cc:407] Connection header received
0927 00:57:16.011126 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:16.011130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:16.011256 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:16.011319 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:16.012027 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48712: BlockingRecv error: recv got EOF from 127.0.0.1:48712 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 00:57:16.035238 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:16.033042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.0.98.61:41657)
0927 00:57:16.033198 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:16.033203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:16.033980 (+ 777us) server_negotiation.cc:407] Connection header received
0927 00:57:16.034224 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:16.034228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:16.034290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:16.034346 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:16.035014 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48728: BlockingRecv error: recv got EOF from 127.0.0.1:48728 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:57:16.072006 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:16.070666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.0.98.61:41657)
0927 00:57:16.070820 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:16.070826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:16.071041 (+ 215us) server_negotiation.cc:407] Connection header received
0927 00:57:16.071217 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:16.071220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:16.071278 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:16.071350 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:16.071856 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48730: BlockingRecv error: recv got EOF from 127.0.0.1:48730 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:57:16.140353 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:16.138717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.0.98.61:41657)
0927 00:57:16.138945 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:16.138950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:16.139169 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:57:16.139408 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:16.139411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:16.139472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:16.139592 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:16.140218 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48744: BlockingRecv error: recv got EOF from 127.0.0.1:48744 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260927 00:57:16.272704 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:16.271381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.0.98.61:41657)
0927 00:57:16.271586 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:16.271591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:16.271746 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:57:16.271888 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:16.271891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:16.271945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:16.272035 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:16.272575 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48752: BlockingRecv error: recv got EOF from 127.0.0.1:48752 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 00:57:16.532794 22800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:16.531649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.98.61:41657)
0927 00:57:16.531883 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:16.531889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:16.531908 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:57:16.532087 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:16.532090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:16.532139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:16.532208 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:16.532672 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48766: BlockingRecv error: recv got EOF from 127.0.0.1:48766 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 00:57:16.848268 22921 catalog_manager.cc:1259] Loaded cluster ID: 6c8e756ae9ff41c496ee1d9f5d8efeb9
I20260927 00:57:16.848354 22921 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: loading cluster ID for follower catalog manager: success
I20260927 00:57:16.850021 22921 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: acquiring CA information for follower catalog manager: success
I20260927 00:57:16.851214 22921 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c87105e9b8d4a58b48a485b17315e47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:57:16.857385 22904 catalog_manager.cc:1259] Loaded cluster ID: 6c8e756ae9ff41c496ee1d9f5d8efeb9
I20260927 00:57:16.857460 22904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: loading cluster ID for follower catalog manager: success
I20260927 00:57:16.859055 22904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: acquiring CA information for follower catalog manager: success
I20260927 00:57:16.860194 22904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74c0128e97c044fc8a5cf89436e3bc66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:57:17.052328 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:17.047440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.0.98.61:41657)
0927 00:57:17.047785 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:17.047797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:17.047816 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:57:17.048043 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:17.048046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:17.048189 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:17.048309 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:17.048673 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.049276 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.050487 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.050735 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.050836 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:17.051350 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:17.051858 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:17.051959 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:17.052097 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:57:17.052155 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
I20260927 00:57:17.060813 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:17.056263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.0.98.60:40541)
0927 00:57:17.056513 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:17.056517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:17.056533 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:57:17.056657 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:17.056659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:17.056828 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:17.056974 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:17.057281 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.057896 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.058892 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.059184 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.059280 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:17.059702 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:17.060345 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:17.060459 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:17.060576 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 00:57:17.060644 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260927 00:57:17.069159 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:17.064880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.0.98.61:41657)
0927 00:57:17.065071 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:17.065076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:17.065351 (+ 275us) server_negotiation.cc:407] Connection header received
0927 00:57:17.065423 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:17.065426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:17.065491 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:17.065578 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:17.066117 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.066712 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.067891 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.068245 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.068355 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:17.068613 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:17.068718 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:17.068782 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:17.068903 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 00:57:17.068972 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:57:17.069953 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:17.064783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.0.98.62:34947)
0927 00:57:17.065097 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:17.065102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:17.065323 (+ 221us) server_negotiation.cc:407] Connection header received
0927 00:57:17.065553 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:17.065556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:17.065777 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:17.065914 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:17.066152 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.067009 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.068308 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.068669 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.068763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:17.069086 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:17.069490 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:17.069577 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:17.069703 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:57:17.069789 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260927 00:57:17.070179 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:17.065179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.0.98.60:40541)
0927 00:57:17.065316 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:17.065320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:17.065681 (+ 361us) server_negotiation.cc:407] Connection header received
0927 00:57:17.065874 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:17.065876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:17.065927 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:17.065994 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:17.066313 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.067106 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.068346 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:17.068630 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:17.068704 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:17.069682 (+ 978us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:17.069772 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:17.069834 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:17.069966 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 00:57:17.070049 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:57:17.073690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22653
I20260927 00:57:17.081386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22728
I20260927 00:57:17.088378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 22802
2026-09-27T00:57:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3101 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:57:17 dist-test-slave-9fh2 krb5kdc[23019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:57:17 dist-test-slave-9fh2 krb5kdc[23019](info): set up 2 sockets
Sep 27 00:57:17 dist-test-slave-9fh2 krb5kdc[23019](info): commencing operation
krb5kdc: starting...
W20260927 00:57:18.059943 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/test-user.keytab.
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:57:18.190147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41599,127.0.98.61:40709,127.0.98.60:33685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:57:18.303486 23032 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:57:18.303823 23032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:18.303848 23032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:18.303887 23032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:18.308386 23032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:18.308475 23032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:18.308493 23032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:18.308509 23032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:18.308524 23032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:18.314482 23032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41599,127.0.98.61:40709,127.0.98.60:33685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:18.315661 23032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:18.316764 23032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:18.326793 23040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:18.326783 23037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:18.326793 23038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:18.328303 23032 server_base.cc:1034] running on GCE node
I20260927 00:57:18.329203 23032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:18.330844 23032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:18.332077 23032 hybrid_clock.cc:648] HybridClock initialized: now 1790470638332061 us; error 83 us; skew 500 ppm
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:18.335793 23032 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:57:18.337085 23032 webserver.cc:460] Webserver started at http://127.0.98.62:41841/ using document root <none> and password file <none>
I20260927 00:57:18.337810 23032 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:18.337898 23032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:18.338142 23032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:18.340839 23032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data/instance:
uuid: "fde58ff0823241ba86081d21655e0ce7"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "e304101343512833deabc8421070374e"
server_key_iv: "6c3d656b166758fe2c38e65e0263e973"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.341594 23032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal/instance:
uuid: "fde58ff0823241ba86081d21655e0ce7"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "e304101343512833deabc8421070374e"
server_key_iv: "6c3d656b166758fe2c38e65e0263e973"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.346762 23032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:57:18.350788 23047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:18.352694 23032 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:57:18.352895 23032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal
uuid: "fde58ff0823241ba86081d21655e0ce7"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "e304101343512833deabc8421070374e"
server_key_iv: "6c3d656b166758fe2c38e65e0263e973"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.353057 23032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:18.392652 23032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:18.396934 23032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:18.397189 23032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:18.405836 23032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41599
I20260927 00:57:18.406144 23099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41599 every 8 connection(s)
I20260927 00:57:18.407101 23032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:57:18.407805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 23032
I20260927 00:57:18.407941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-0/wal/instance
I20260927 00:57:18.408258 392 external_mini_cluster.cc:1428] Setting key c92e3a39697b0219f481e2683a5a1d64
I20260927 00:57:18.410876 23100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:18.415027 23100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:18.434479 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.417735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.98.62:41599)
0927 00:57:18.418233 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.418250 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.418686 (+ 436us) server_negotiation.cc:407] Connection header received
0927 00:57:18.420363 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.420370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.420695 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.420934 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.422158 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.423223 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.424732 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.425215 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.425351 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.428777 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.428784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.428799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.428834 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.430586 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.431322 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.431328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.431329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.431472 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.431925 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.431928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.431930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.432428 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.432559 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.433133 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.433403 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":146,"threads_started":1}
I20260927 00:57:18.434487 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.417736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41599 (local address 127.0.0.1:56354)
0927 00:57:18.418409 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.418477 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.419884 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.420997 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.421004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.421477 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.421939 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.421961 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.423382 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.423386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.424576 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.424588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.425422 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.425426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.425577 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.426282 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.426310 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.428513 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.430732 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.430743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.430756 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.431116 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.431611 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.431615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.431617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.431770 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.432580 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.432590 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.432792 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.432986 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.433559 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":106,"spinlock_wait_cycles":58112,"thread_start_us":217,"threads_started":1}
I20260927 00:57:18.437954 23100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:57:18.438846 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.438465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45520)
0927 00:57:18.438646 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.438751 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 00:57:18.439092 23051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
I20260927 00:57:18.439287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40709
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41599,127.0.98.61:40709,127.0.98.60:33685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:57:18.440478 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.440192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45524)
0927 00:57:18.440341 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.440398 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 00:57:18.440706 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
I20260927 00:57:18.490082 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } attempt: 1
I20260927 00:57:18.491393 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.490905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45532)
0927 00:57:18.491137 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.491273 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 00:57:18.492385 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.492121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45544)
0927 00:57:18.492251 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.492314 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 00:57:18.492573 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
I20260927 00:57:18.560768 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } attempt: 2
I20260927 00:57:18.561825 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.561406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45548)
0927 00:57:18.561617 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.561712 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 00:57:18.562608 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.562375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45550)
0927 00:57:18.562492 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.562545 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 00:57:18.562772 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40709: connect: Connection refused (error 111)
W20260927 00:57:18.572541 23107 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:57:18.572906 23107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:18.572938 23107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:18.572993 23107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:18.577502 23107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:18.577610 23107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:18.577630 23107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:18.577647 23107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:18.577662 23107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:18.583772 23107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41599,127.0.98.61:40709,127.0.98.60:33685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:18.585235 23107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:18.586541 23107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:18.597224 23113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:18.597245 23112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:18.597245 23115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:18.597625 23107 server_base.cc:1034] running on GCE node
I20260927 00:57:18.598328 23107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:18.599864 23107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:18.601063 23107 hybrid_clock.cc:648] HybridClock initialized: now 1790470638601042 us; error 48 us; skew 500 ppm
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:18.604362 23107 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:57:18.605638 23107 webserver.cc:460] Webserver started at http://127.0.98.61:39771/ using document root <none> and password file <none>
I20260927 00:57:18.606319 23107 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:18.606398 23107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:18.606634 23107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:18.609206 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data/instance:
uuid: "9aea2d5cc8234d99992084214d371d0a"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "6b13772b84107a9fa9c06c7198b9299e"
server_key_iv: "ba5a53fdc769ed122b1982e191b36d73"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.609894 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal/instance:
uuid: "9aea2d5cc8234d99992084214d371d0a"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "6b13772b84107a9fa9c06c7198b9299e"
server_key_iv: "ba5a53fdc769ed122b1982e191b36d73"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.614722 23107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 00:57:18.618223 23122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:18.620040 23107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:18.620239 23107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal
uuid: "9aea2d5cc8234d99992084214d371d0a"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "6b13772b84107a9fa9c06c7198b9299e"
server_key_iv: "ba5a53fdc769ed122b1982e191b36d73"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.620388 23107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:18.636539 23107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:18.640372 23107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:18.640607 23107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:18.649752 23107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40709
I20260927 00:57:18.650064 23174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40709 every 8 connection(s)
I20260927 00:57:18.650996 23107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:57:18.653945 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } attempt: 3
I20260927 00:57:18.654125 23175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:18.657357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 23107
I20260927 00:57:18.657496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-1/wal/instance
I20260927 00:57:18.657842 392 external_mini_cluster.cc:1428] Setting key 41395d01ae3a50b583ea465bb29303b4
I20260927 00:57:18.660149 23175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:57:18.669709 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.654623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45560)
0927 00:57:18.654823 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.654845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.654977 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.658271 (+ 3294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.658275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.658557 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.658724 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.658732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.660590 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.660596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.661453 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.661468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.662541 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.662544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.662687 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.663199 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.663212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.664638 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.667370 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.667373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.667375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.667608 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.668082 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.668087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.668089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.668209 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.669095 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.669100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.669243 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.669371 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.669542 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 00:57:18.670791 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.655477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.0.98.61:40709)
0927 00:57:18.656003 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.656073 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.656122 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:57:18.657205 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.657228 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.657880 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.658321 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.658869 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.660373 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.661643 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.662410 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.662584 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.664851 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.664859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.664875 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.664918 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.667236 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.667732 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.667739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.667747 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.667924 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.668344 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.668349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.668351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.668860 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.668978 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.669513 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.669883 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":167,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:18.673853 23100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:18.674947 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.674471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:60998)
0927 00:57:18.674758 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.674838 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 00:57:18.676604 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.675588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32778)
0927 00:57:18.676411 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.676469 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":564,"threads_started":1}
I20260927 00:57:18.677170 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.662151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41599 (local address 127.0.0.1:56358)
0927 00:57:18.663088 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.663180 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.663329 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.663682 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.663688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.664250 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.664457 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.664466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.665535 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.665538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.666350 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.666364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.666855 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.666859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.666974 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.667817 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.667831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.672259 (+ 4428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.674988 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.675001 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.675009 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.675423 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.675795 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.675798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.675800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.675924 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.676509 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.676517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.676682 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.676844 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.677020 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"spinlock_wait_cycles":6528,"thread_start_us":113,"threads_started":1}
W20260927 00:57:18.677362 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33685: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
I20260927 00:57:18.677954 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.662235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.0.98.62:41599)
0927 00:57:18.662404 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.662408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.663288 (+ 880us) server_negotiation.cc:407] Connection header received
0927 00:57:18.663437 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.663440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.663493 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.663585 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.664606 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.665170 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.666475 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.666737 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.666824 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.672451 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.672456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.672459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.672495 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.674826 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.675569 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.675574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.675576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.675679 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.676046 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.676049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.676051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.676325 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.676481 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.676943 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.677104 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:57:18.679392 23175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:57:18.688294 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.679970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45572)
0927 00:57:18.680106 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.680120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.680213 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.680432 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.680435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.680613 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.680707 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.680713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.681442 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.681445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.682009 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.682017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.682430 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.682433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.682511 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.683038 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.683051 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.684487 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.686480 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.686485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.686488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.686803 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.687118 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.687121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.687123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.687273 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.687748 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.687752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.687872 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.687967 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.688153 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 00:57:18.688410 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.680022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.0.98.61:40709)
0927 00:57:18.680119 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.680124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.680187 (+ 63us) server_negotiation.cc:407] Connection header received
0927 00:57:18.680295 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.680297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.680345 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.680405 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.680806 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.681342 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.682108 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.682329 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.682414 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.684694 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.684699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.684701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.684739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.686304 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.686910 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.686913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.686914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.686999 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.687398 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.687401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.687403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.687632 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.687719 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.688048 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.688223 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":49}
I20260927 00:57:18.689422 23175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:18.690418 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.689967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32790)
0927 00:57:18.690224 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.690337 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab.
W20260927 00:57:18.691116 23123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
I20260927 00:57:18.692214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
/tmp/dist-test-taskj1hhZs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33685
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41599,127.0.98.61:40709,127.0.98.60:33685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:57:18.692319 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.692051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32794)
0927 00:57:18.692187 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.692242 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 00:57:18.692734 23175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33685: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
I20260927 00:57:18.709044 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } attempt: 1
I20260927 00:57:18.710242 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.709794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32806)
0927 00:57:18.710000 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.710136 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 00:57:18.711045 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.710820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32810)
0927 00:57:18.710926 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.710979 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:57:18.711205 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33685: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
I20260927 00:57:18.742060 23175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } attempt: 1
I20260927 00:57:18.743420 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.742820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32818)
0927 00:57:18.743098 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.743302 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 00:57:18.744434 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.744167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32820)
0927 00:57:18.744293 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.744344 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:57:18.744587 23175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33685: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
I20260927 00:57:18.786398 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } attempt: 2
I20260927 00:57:18.787572 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.787191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32836)
0927 00:57:18.787376 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.787443 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 00:57:18.788697 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.788421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32848)
0927 00:57:18.788552 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.788629 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:57:18.788862 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33685: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
I20260927 00:57:18.812765 23175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } attempt: 2
I20260927 00:57:18.813791 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.813460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32864)
0927 00:57:18.813633 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.813703 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:57:18.814780 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.814522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32868)
0927 00:57:18.814658 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.814712 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 00:57:18.814935 23175 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33685: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
W20260927 00:57:18.815364 23183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:57:18.815670 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:57:18.815716 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:57:18.815790 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:57:18.819998 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:57:18.820086 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:57:18.820104 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:57:18.820120 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:57:18.820135 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:57:18.826134 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41599,127.0.98.61:40709,127.0.98.60:33685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:57:18.827340 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:57:18.828413 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:57:18.837505 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:18.837630 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:57:18.837541 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:57:18.837965 23183 server_base.cc:1034] running on GCE node
I20260927 00:57:18.838641 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:57:18.839923 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:57:18.841112 23183 hybrid_clock.cc:648] HybridClock initialized: now 1790470638841076 us; error 57 us; skew 500 ppm
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:57:18.843961 23183 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:57:18.844866 23183 webserver.cc:460] Webserver started at http://127.0.98.60:32953/ using document root <none> and password file <none>
I20260927 00:57:18.845453 23183 fs_manager.cc:359] Metadata directory not provided
I20260927 00:57:18.845522 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:57:18.845744 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:57:18.848002 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data/instance:
uuid: "d3578d399ea24c20864cf0b038f8e40f"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "2fb33ff61df7d6165f789275a09eb092"
server_key_iv: "58ce0af5247588894a389e0ddc6f2865"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.848555 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal/instance:
uuid: "d3578d399ea24c20864cf0b038f8e40f"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "2fb33ff61df7d6165f789275a09eb092"
server_key_iv: "58ce0af5247588894a389e0ddc6f2865"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.852746 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:57:18.855448 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:57:18.857017 23183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:57:18.857172 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal
uuid: "d3578d399ea24c20864cf0b038f8e40f"
format_stamp: "Formatted at 2026-09-27 00:57:18 on dist-test-slave-9fh2"
server_key: "2fb33ff61df7d6165f789275a09eb092"
server_key_iv: "58ce0af5247588894a389e0ddc6f2865"
server_key_version: "encryptionkey@0"
I20260927 00:57:18.857306 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:57:18.873840 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:57:18.881358 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:57:18.881567 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:57:18.888015 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } attempt: 3
I20260927 00:57:18.889061 23183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33685
I20260927 00:57:18.889148 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.888700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32870)
0927 00:57:18.888930 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.889028 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 00:57:18.889477 23250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33685 every 8 connection(s)
I20260927 00:57:18.890292 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:57:18.893764 23251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:57:18.898187 23251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:57:18.900085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu as pid 23183
I20260927 00:57:18.900244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj1hhZs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470348013064-392-0/minicluster-data/master-2/wal/instance
I20260927 00:57:18.900547 392 external_mini_cluster.cc:1428] Setting key 059915dc37ddfc3c7552b85f8ab49ab8
I20260927 00:57:18.906116 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.889914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32886)
0927 00:57:18.890073 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.890099 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.890225 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.893290 (+ 3065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.893293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.893561 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.893742 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.893750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.895657 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.895661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.896329 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.896344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.897460 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.897464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.897564 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.898240 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.898254 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.900030 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.903468 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.903472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.903474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.903788 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.904232 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.904235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.904237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.904340 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.905485 (+ 1145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.905489 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.905624 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.905730 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.905909 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:57:18.906111 23175 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } attempt: 3
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:57:18.908499 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.890907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.98.60:33685)
0927 00:57:18.891503 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.891523 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.891569 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:57:18.892457 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.892479 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.892920 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.893300 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.893937 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.895473 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.896513 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.897238 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.897416 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.900537 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.900546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.900574 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.900686 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.903304 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.903921 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.903928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.903936 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.904095 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.904457 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.904461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.904463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.905053 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.905181 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.907042 (+ 1861us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.907418 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":197,"threads_started":1}
I20260927 00:57:18.911943 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.900196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41599 (local address 127.0.0.1:56362)
0927 00:57:18.900759 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.900839 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.901154 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.901578 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.901585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.902112 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.902296 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.902305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.903475 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.903477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.904321 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.904328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.904780 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.904784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.904920 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.905502 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.905513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.907656 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.909823 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.909833 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.909841 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.910187 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.910539 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.910542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.910545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.910648 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.911218 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.911225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.911363 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.911501 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.911738 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":174,"threads_started":1}
I20260927 00:57:18.912984 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.900248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.0.98.62:41599)
0927 00:57:18.900401 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.900405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.901033 (+ 628us) server_negotiation.cc:407] Connection header received
0927 00:57:18.901335 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.901340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.901409 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.901500 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.902495 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.903340 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.904434 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.904666 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.904757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.907872 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.907877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.907880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.907906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.909671 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.910316 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.910320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.910322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.910424 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.910755 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.910759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.910760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.911087 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.912634 (+ 1547us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.912712 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.912834 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:57:18.913287 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.901674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.0.98.60:33685)
0927 00:57:18.902036 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.902047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.902059 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:57:18.902090 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.902092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.902130 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.902270 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.903085 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.903831 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.904900 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.905228 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.905317 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.908303 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.908308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.908310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.908336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.910997 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.911577 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.911580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.911582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.911663 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.911951 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.911954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.911956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.912178 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.912335 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.912999 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.913136 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:57:18.914165 23251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:18.914986 23100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7: Bootstrap starting.
I20260927 00:57:18.918265 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.906761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32898)
0927 00:57:18.906932 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.906950 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.907038 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.908548 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.908552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.909105 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.909252 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.909260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.910594 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.910598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.911424 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.911435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.912002 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.912006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.912091 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.912556 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.912569 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.914208 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.916509 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.916512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.916515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.916898 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.917208 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.917211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.917213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.917301 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.917735 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.917738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.917862 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.917946 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.918097 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":19}
I20260927 00:57:18.918810 23100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:18.918915 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.907035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.0.98.60:33685)
0927 00:57:18.907987 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.907996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.908009 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:57:18.908043 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.908047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.908094 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.909360 (+ 1266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.909491 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.910474 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.911562 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.911868 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.913157 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.914628 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.914632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.914635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.914659 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.916383 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.917006 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.917010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.917012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.917108 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.917397 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.917400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.917402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.917632 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.918526 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.918622 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.918743 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":132,"threads_started":1}
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:57:18.920092 23100 log.cc:824] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:18.922158 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: Bootstrap starting.
I20260927 00:57:18.922833 23100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7: No bootstrap required, opened a new log
I20260927 00:57:18.924129 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.914695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40709 (local address 127.0.0.1:45588)
0927 00:57:18.915189 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.915203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.915282 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.915678 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.915681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.915868 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.915950 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.915961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.916824 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.916827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.917617 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.917625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.918263 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.918267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.918344 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.918808 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.918821 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.920394 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.922348 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.922351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.922353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.922592 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.922882 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.922885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.922887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.923062 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.923622 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.923625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.923725 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.923826 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.923965 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398}
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:57:18.924721 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.914945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.0.98.61:40709)
0927 00:57:18.915073 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.915077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.915361 (+ 284us) server_negotiation.cc:407] Connection header received
0927 00:57:18.915407 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.915410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.915462 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.915531 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.916075 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.916691 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.917735 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.918057 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.918154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.920543 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.920547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.920550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.920575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.922161 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.922702 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.922705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.922707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.922788 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.923222 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.923225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.923227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.923480 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.923574 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.924469 (+ 895us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.924583 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260927 00:57:18.925436 23175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:18.926347 23175 log.cc:824] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:18.926524 23251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:57:18.928323 23100 raft_consensus.cc:348] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.928627 23100 raft_consensus.cc:729] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde58ff0823241ba86081d21655e0ce7, State: Initialized, Role: FOLLOWER
I20260927 00:57:18.929059 23100 consensus_queue.cc:260] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.929152 23175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: No bootstrap required, opened a new log
I20260927 00:57:18.929730 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.915414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.62:41599)
0927 00:57:18.916692 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.916696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.916710 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:57:18.916754 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.916757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.916815 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.916897 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.917818 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.918782 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.921658 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.921972 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.922072 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.925171 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.925175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.925178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.925205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.927268 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.927784 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.927787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.927789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.927874 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.928145 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.928148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.928150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.928395 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.928770 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.929440 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.929570 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086}
I20260927 00:57:18.931108 23100 sys_catalog.cc:564] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:57:18.933454 23175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.933748 23175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aea2d5cc8234d99992084214d371d0a, State: Initialized, Role: FOLLOWER
I20260927 00:57:18.934105 23175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.935664 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.927043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33685 (local address 127.0.0.1:32908)
0927 00:57:18.927270 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 00:57:18.927286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:57:18.927364 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:57:18.927610 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:57:18.927612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:57:18.927782 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:57:18.927864 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.927870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.928790 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.928793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.929512 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:57:18.929522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.930165 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.930168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:57:18.930240 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.930689 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:57:18.930703 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:57:18.932332 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:57:18.934221 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.934224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.934226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.934442 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.934688 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:57:18.934690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:57:18.934692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:57:18.934759 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:57:18.935187 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:57:18.935190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:57:18.935271 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 00:57:18.935369 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:57:18.935500 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 00:57:18.935727 23175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:18.936245 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.927057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.98.60:33685)
0927 00:57:18.927435 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.927439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.927450 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:57:18.927477 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.927479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.927516 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.927576 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.927973 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.928683 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.929636 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.929953 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.930045 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.932476 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.932480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.932483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.932504 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.934105 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.934537 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.934539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.934541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.934611 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.934829 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.934831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.934833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.935025 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.935122 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.935999 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.936120 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
W20260927 00:57:18.938105 23271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:18.938210 23271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:18.939172 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: Bootstrap starting.
Sep 27 00:57:18 dist-test-slave-9fh2 krb5kdc[23019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:57:18.942118 23251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: Neither blocks nor log segments found. Creating new log.
I20260927 00:57:18.938335 23032 master_runner.cc:186] Error getting master registration for 127.0.98.61:40709: OK, instance_id { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" instance_seqno: 1790470638647089 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:57:18.942505 23032 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:57:18.943008 23251 log.cc:824] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:57:18.945883 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: No bootstrap required, opened a new log
I20260927 00:57:18.946931 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.931723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.98.61:40709)
0927 00:57:18.931925 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.931930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.932215 (+ 285us) server_negotiation.cc:407] Connection header received
0927 00:57:18.932445 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.932449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.932504 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.932564 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.933504 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.934270 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.938683 (+ 4413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.939002 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.939079 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.942776 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.942780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.942783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.942821 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.944595 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.945222 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.945225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.945227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.945309 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.945587 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.945590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.945592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.945808 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.945956 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.946676 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.946787 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 00:57:18.947396 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:18.947588 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:18.947904 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:18.948060 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:18.950425 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:57:18.951577 23074 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fde58ff0823241ba86081d21655e0ce7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56366
I20260927 00:57:18.951825 23074 raft_consensus.cc:482] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:57:18.951922 23074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:18.953403 23074 raft_consensus.cc:504] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.953912 23251 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.954214 23251 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3578d399ea24c20864cf0b038f8e40f, State: Initialized, Role: FOLLOWER
I20260927 00:57:18.954401 23074 leader_election.cc:288] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 9aea2d5cc8234d99992084214d371d0a (127.0.98.61:40709), d3578d399ea24c20864cf0b038f8e40f (127.0.98.60:33685)
I20260927 00:57:18.951000 23107 master_runner.cc:186] Error getting master registration for 127.0.98.60:33685: OK, instance_id { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" instance_seqno: 1790470638886880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:57:18.954700 23107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:57:18.954645 23251 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.956406 23251 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:57:18.956777 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fde58ff0823241ba86081d21655e0ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3578d399ea24c20864cf0b038f8e40f"
I20260927 00:57:18.957098 23225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:18.958544 23225 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde58ff0823241ba86081d21655e0ce7 in term 1.
I20260927 00:57:18.959014 23048 leader_election.cc:302] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3578d399ea24c20864cf0b038f8e40f, fde58ff0823241ba86081d21655e0ce7; no voters:
W20260927 00:57:18.959261 23285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:18.959388 23285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:18.959379 23260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:57:18.959658 23260 raft_consensus.cc:686] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [term 1 LEADER]: Becoming Leader. State: Replica: fde58ff0823241ba86081d21655e0ce7, State: Running, Role: LEADER
I20260927 00:57:18.959713 23149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fde58ff0823241ba86081d21655e0ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9aea2d5cc8234d99992084214d371d0a"
I20260927 00:57:18.959940 23149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:57:18.960148 23260 consensus_queue.cc:237] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } }
I20260927 00:57:18.961213 23149 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde58ff0823241ba86081d21655e0ce7 in term 1.
W20260927 00:57:18.965355 23304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:57:18.965507 23304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:57:18.967751 23302 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fde58ff0823241ba86081d21655e0ce7. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:18.967950 23302 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:18.969906 23307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:57:18.971045 23307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:57:18.972699 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.959125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.0.98.60:33685)
0927 00:57:18.959332 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.959338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.960201 (+ 863us) server_negotiation.cc:407] Connection header received
0927 00:57:18.960447 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.960450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.960511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.960614 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.961778 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.962592 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.965514 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.965866 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.965942 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.967997 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.968002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.968004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.968034 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.969729 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.970259 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.970264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.970266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.970387 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.970684 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.970687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.970689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.970944 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.971099 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.972359 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.972514 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 00:57:18.978672 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.968702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.0.98.62:41599)
0927 00:57:18.968822 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.968826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.970089 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 00:57:18.970284 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.970287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.970339 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.970389 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.971349 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.972073 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.973162 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.973451 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.973521 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.975242 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.975246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.975248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.975275 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.976804 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.977247 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.977250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.977252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.977335 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.977651 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.977654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.977656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.977882 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.978008 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.978363 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.978485 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:57:18.982031 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:18.969388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.98.61:40709)
0927 00:57:18.969519 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:18.969523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:18.972010 (+ 2487us) server_negotiation.cc:407] Connection header received
0927 00:57:18.972064 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:18.972067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:18.972126 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:18.972197 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:18.974586 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.975364 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.976917 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:18.977248 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:18.977335 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:18.978764 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:57:18.978767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:57:18.978770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:57:18.978797 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:57:18.980555 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.980942 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.980946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.980947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.981021 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:57:18.981256 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:57:18.981260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:57:18.981262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:57:18.981438 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:57:18.981565 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:18.981777 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:57:18.981881 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:57:18.983430 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:18.983680 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:18.995779 23225 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [term 1 FOLLOWER]: Refusing update from remote peer fde58ff0823241ba86081d21655e0ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:57:18.996012 23149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [term 1 FOLLOWER]: Refusing update from remote peer fde58ff0823241ba86081d21655e0ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:57:18.996779 23260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:57:18.997392 23302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:57:19.001870 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fde58ff0823241ba86081d21655e0ce7. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:19.002100 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:19.002252 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fde58ff0823241ba86081d21655e0ce7. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:19.002383 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:19.009825 23302 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:19.010057 23302 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:19.011766 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:19.011950 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:19.013330 23310 mvcc.cc:205] Tried to move back new op lower bound from 7333767737302323200 to 7333767737186209792. Current Snapshot: MvccSnapshot[applied={T|T < 7333767737302323200}]
I20260927 00:57:19.015461 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:19.015641 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:57:19.017247 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fde58ff0823241ba86081d21655e0ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde58ff0823241ba86081d21655e0ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } peers { permanent_uuid: "9aea2d5cc8234d99992084214d371d0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40709 } } peers { permanent_uuid: "d3578d399ea24c20864cf0b038f8e40f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33685 } } }
I20260927 00:57:19.017421 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:57:19.018049 23312 mvcc.cc:205] Tried to move back new op lower bound from 7333767737302323200 to 7333767737186209792. Current Snapshot: MvccSnapshot[applied={T|T < 7333767737302323200}]
I20260927 00:57:19.018731 23311 mvcc.cc:205] Tried to move back new op lower bound from 7333767737302323200 to 7333767737186209792. Current Snapshot: MvccSnapshot[applied={T|T < 7333767737302323200}]
I20260927 00:57:19.021374 23307 catalog_manager.cc:1347] Generated new cluster ID: 05990f1b0b98480fae2aa3c73ae4bab6
I20260927 00:57:19.021474 23307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:57:19.076748 23307 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:57:19.078325 23307 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:57:19.093997 23307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fde58ff0823241ba86081d21655e0ce7: Generated new TSK 0
I20260927 00:57:19.095387 23307 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:57:19.139870 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.136510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.0.98.62:41599)
0927 00:57:19.136713 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.136718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.138276 (+ 1558us) server_negotiation.cc:407] Connection header received
0927 00:57:19.138744 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.138748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.138815 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.138886 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.139683 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56386: BlockingRecv error: recv got EOF from 127.0.0.1:56386 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 00:57:19.141474 23257 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.136036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.0.98.60:33685)
0927 00:57:19.136186 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.136192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.138643 (+ 2451us) server_negotiation.cc:407] Connection header received
0927 00:57:19.140523 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.140527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.140593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.140654 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.141313 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32938: BlockingRecv error: recv got EOF from 127.0.0.1:32938 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:57:19.144809 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.137338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.0.98.61:40709)
0927 00:57:19.137475 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.137480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.142253 (+ 4773us) server_negotiation.cc:407] Connection header received
0927 00:57:19.143886 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.143891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.143959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.144023 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.144649 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45632: BlockingRecv error: recv got EOF from 127.0.0.1:45632 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 00:57:19.152568 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.149108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.0.98.61:40709)
0927 00:57:19.149268 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.149273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.151249 (+ 1976us) server_negotiation.cc:407] Connection header received
0927 00:57:19.151562 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.151566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.151673 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.151742 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.152422 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45642: BlockingRecv error: recv got EOF from 127.0.0.1:45642 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:57:19.159030 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.157208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.98.61:40709)
0927 00:57:19.157380 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.157386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.157676 (+ 290us) server_negotiation.cc:407] Connection header received
0927 00:57:19.157894 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.157898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.157997 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.158141 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.158864 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45654: BlockingRecv error: recv got EOF from 127.0.0.1:45654 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 00:57:19.168742 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.163760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.0.98.61:40709)
0927 00:57:19.163893 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.163897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.167593 (+ 3696us) server_negotiation.cc:407] Connection header received
0927 00:57:19.167753 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.167757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.167850 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.167984 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.168564 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45656: BlockingRecv error: recv got EOF from 127.0.0.1:45656 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:57:19.176949 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.175487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.0.98.61:40709)
0927 00:57:19.175719 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.175725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.175918 (+ 193us) server_negotiation.cc:407] Connection header received
0927 00:57:19.176058 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.176063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.176146 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.176277 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.176806 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45662: BlockingRecv error: recv got EOF from 127.0.0.1:45662 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
W20260927 00:57:19.189556 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.188227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.0.98.61:40709)
0927 00:57:19.188384 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.188389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.188577 (+ 188us) server_negotiation.cc:407] Connection header received
0927 00:57:19.188749 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.188753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.188817 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.188899 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.189409 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45664: BlockingRecv error: recv got EOF from 127.0.0.1:45664 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 00:57:19.209834 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.208346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.98.61:40709)
0927 00:57:19.208590 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.208596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.208755 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:57:19.209000 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.209003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.209061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.209141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.209691 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45678: BlockingRecv error: recv got EOF from 127.0.0.1:45678 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260927 00:57:19.246464 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.244541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.98.61:40709)
0927 00:57:19.244737 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.244742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.245097 (+ 355us) server_negotiation.cc:407] Connection header received
0927 00:57:19.245337 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.245353 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.245442 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.245573 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.246285 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45692: BlockingRecv error: recv got EOF from 127.0.0.1:45692 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 00:57:19.314837 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.313320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.0.98.61:40709)
0927 00:57:19.313508 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.313513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.313674 (+ 161us) server_negotiation.cc:407] Connection header received
0927 00:57:19.313857 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.313860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.313915 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.313990 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.314682 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45698: BlockingRecv error: recv got EOF from 127.0.0.1:45698 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 00:57:19.447028 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.445447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.98.61:40709)
0927 00:57:19.445703 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.445707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.445842 (+ 135us) server_negotiation.cc:407] Connection header received
0927 00:57:19.446126 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.446131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.446215 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.446308 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.446873 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45704: BlockingRecv error: recv got EOF from 127.0.0.1:45704 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260927 00:57:19.707057 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:57:19.705667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.0.98.61:40709)
0927 00:57:19.705968 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:19.705973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:19.705994 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:57:19.706197 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:19.706201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:19.706270 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:19.706365 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:57:19.706899 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45714: BlockingRecv error: recv got EOF from 127.0.0.1:45714 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 00:57:19.961118 23285 catalog_manager.cc:1259] Loaded cluster ID: 05990f1b0b98480fae2aa3c73ae4bab6
I20260927 00:57:19.961230 23285 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: loading cluster ID for follower catalog manager: success
I20260927 00:57:19.963049 23285 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: acquiring CA information for follower catalog manager: success
I20260927 00:57:19.964382 23285 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9aea2d5cc8234d99992084214d371d0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:57:19.966856 23304 catalog_manager.cc:1259] Loaded cluster ID: 05990f1b0b98480fae2aa3c73ae4bab6
I20260927 00:57:19.966928 23304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: loading cluster ID for follower catalog manager: success
I20260927 00:57:19.968602 23304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: acquiring CA information for follower catalog manager: success
I20260927 00:57:19.969761 23304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3578d399ea24c20864cf0b038f8e40f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:57:20.226673 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:20.221760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.0.98.61:40709)
0927 00:57:20.222067 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:20.222071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:20.222090 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:57:20.222259 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:20.222262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:20.222427 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:20.222574 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:20.222940 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.223605 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.224590 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.224866 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.224963 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:20.225328 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:20.226008 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:20.226138 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:20.226328 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 00:57:20.226414 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260927 00:57:20.236022 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:20.231036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.98.60:33685)
0927 00:57:20.231368 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:20.231373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:20.231391 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:57:20.231632 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:20.231635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:20.231796 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:20.232007 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:20.232260 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.232849 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.233835 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.234109 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.234203 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:20.234623 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:20.235444 (+ 821us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:20.235594 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:20.235686 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 00:57:20.235771 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260927 00:57:20.245419 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:20.240585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.0.98.61:40709)
0927 00:57:20.240727 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:20.240732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:20.240971 (+ 239us) server_negotiation.cc:407] Connection header received
0927 00:57:20.241188 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:20.241193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:20.241280 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:20.241349 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:20.241716 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.242330 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.243864 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.244205 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.244305 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:20.244570 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:20.244717 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:20.244785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:20.245176 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 00:57:20.245270 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:57:20.245453 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:20.240127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.0.98.62:41599)
0927 00:57:20.240420 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:20.240424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:20.240681 (+ 257us) server_negotiation.cc:407] Connection header received
0927 00:57:20.241294 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:20.241298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:20.241494 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:20.241649 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:20.241871 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.242634 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.243643 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.243926 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.244028 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:20.244438 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:20.244903 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:20.244978 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:20.245111 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 00:57:20.245188 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":69,"threads_started":1}
I20260927 00:57:20.245591 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:57:20.240312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.98.60:33685)
0927 00:57:20.240443 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:57:20.240448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:57:20.240918 (+ 470us) server_negotiation.cc:407] Connection header received
0927 00:57:20.241068 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:57:20.241071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:57:20.241137 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:57:20.241240 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:57:20.241523 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.242396 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.244391 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:57:20.244709 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:57:20.244813 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:57:20.245078 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:57:20.245167 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:57:20.245235 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:57:20.245394 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 00:57:20.245457 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:57:20.249428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 23032
I20260927 00:57:20.256086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 23107
I20260927 00:57:20.262702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj1hhZs/build/debug/bin/kudu with pid 23183
2026-09-27T00:57:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3174 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6276 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292242 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST